42
Mobile Testing Explained: Terms, Phases, Costs, & More
Most users can barely go an hour without their mobile devices, or, rather, the apps that are available on them. Technology going mobile has given a new twist to every aspect of our lives. As our perception of mobility has changed, so have the standards for mobile software development. A successful mobile application in 2021 is expected to not just work smoothly, but take the users’ breath away with out-of-this-world functionality. Otherwise, your fresh app release is at high risk of getting lost in the pile. In this article, we’re breaking down the instrument turning incompetent applications into powerful ones ― mobile software testing.
Numbers to consider:
Software testing is the only way to tell whether a program works as required (or works at all). As the main part of Quality Assurance, app testing is a multi-level process of huge importance when it comes to digital releases. Working in software development for two decades straight, we can’t help but stress the vital importance, cost-effectiveness, and strategic influence of application testing for any kind of solution ― small, large, complex, etc. Here are more reasons why we equate mobile app testing with development, planning, and technical support in terms of importance:
#1. Early testing is cheaper than last-minute fixing
History knows many examples of programmers and/or product owners acting carelessly about software testing. Even though not every bug is headline worthy (like The Great Google Glitch of 2020 or the infamous case of Amazon’s £1 sale), all the small bugs accumulate to trillions worth of financial losses every year. We’re not trying to deny the existence of lucky entrepreneurs, but want you to think: if absolute market leaders incur financial losses due to their technical errors, what can such an error do to a smaller company?
History knows many examples of programmers and/or product owners acting carelessly about software testing. Even though not every bug is headline worthy (like The Great Google Glitch of 2020 or the infamous case of Amazon’s £1 sale), all the small bugs accumulate to trillions worth of financial losses every year. We’re not trying to deny the existence of lucky entrepreneurs, but want you to think: if absolute market leaders incur financial losses due to their technical errors, what can such an error do to a smaller company?
Software mistakes are preventable, and the only techniques to prevent them are quality assurance and in-depth software testing. With professionally executed QA, a potential glitch can be detected long before it takes place in real life. Budgets for QA practices can be easily estimated and planned ahead of time. But who can tell how much you’d need to urgently fix something in your product? This number is completely unpredictable, and considering the urgency and financial damage from the error itself, do not expect to get off lightly. Underestimating software testing means blindly betting the future of all your investments in the product on your developers’ skills and unwavering attention to detail.
According to data collected by Quettra, 77% of people abandon an app in just three days since they install it. This percentage grows to 90% in one month and reaches the 95%-mark in 90 days. This means that a mobile application has roughly 72 hours to impress the user and start forming a habit of them using the app on a regular basis. Obviously, if an application fails to work as required, it is unlikely that users will spend three whole days trying it. Frankly speaking, these days people barely give buggy apps another go after a single crash. Why bother if there are literally millions of other options in app markets?
The fact people quickly lose interest in applications they download can be interpreted differently. For many, the easiest way out seems to be relying on push notifications. If a person installed an app and left it hanging, why not remind them that the program’s still there with an innocent message or two? That would be totally right if push notifications weren’t as overused as they are today. A mobile application can be considered successful only if users demonstrate a sincere willingness to use it regularly without being annoyed by countless pushes. That is absolutely achievable if the right mobile app testing services are applied from the very start of a project.
Another approach we frequently come across in mobile software development is to release a prototype instead of a polished version of a digital product. People doing so tend to think that a faster time to market is more important than the app’s performance. After all, you can always listen to negative feedback and release an update, right? Unfortunately, chances are the improved version of your product won’t get much attention in app stores considering the initial negative experience.
Putting your name on an application with questionable operability is very risky for your brand image and long-term reputation. Prototyping, MVP releases, and many other software development and mobile application testing techniques can keep you safe, so we highly recommend fitting them into your project planning.
Many mobile development and testing teams agree that it’s inaccurate to think of mobile applications as the same software running on a smaller device. Indeed, mobile application testing services differ greatly from any other project type. Here’s how we see the unique traits of mobile software testing.
Mobile software is called mobile for a reason: these applications are expected to work on the go, anywhere, and at any time. Furthermore, accessibility is a key distinctive feature of mobile software. The different physical interactions that users have with their mobile devices change a lot for developers, UI/UX designers, and testers. At the same time, the global trend of digital experience personalization also added its twist to user expectations from the software they choose to install. When personalization meets accessibility requirements, demands for project testing teams can get out of hand.
Given that the average smartphone user checks their phone every 6 minutes and expects an app to launch in under 2 seconds, we can confidently say that user requirements for mobile software are far higher than they are for desktop and web applications. Here’s a short list of quick tips that can help app owners evaluate potential user expectations:
One of the biggest pain points of mobile software testing is fragmentation ― a term used to describe the existence of many different physical devices running on different operating systems. And if the number of mobile operating systems nowadays pretty much ends on iOS and Android, there are still numerous versions of each. Add to this a ton of smartphones and tablets, flagship devices and old-but-still-in-use, and you’ll get countless combinations that influence application performance. This is a challenge for mobile testers: they have to define the most used device models and OS versions and document possible combinations to use as a basis for their testing strategy. Obviously, it is impossible to test every single combination, but using the most prevalent of them as your guidance is necessary.
Considering the complicated nature of testing fragmentation, teams that opt for thorough mobile QA testing usually prefer having actual physical devices on hand. Emulators can be useful, but things like UI/UX and installation are hard to be fully evaluated with them. User perception of the app changes drastically depending on screen size, navigation buttons (or gestures), and other technical characteristics of a gadget. This is incomparable to screen resolution change or a few-inch difference in PCs and laptops. Our recommendation here is to accommodate your mobile app QA team with the necessary number of physical devices and make sure you target more than just the newest mobile platform versions.
Or you can always outsource to a team that already has a wide range of physical devices available.
In addition to fragmentation, your QA team must also consider mobile OS release cycles. Mobile hardware manufacturers are exposed to an unbelievable level of competition in their industry, which forces them to throw innovation at us as frequently as they can. Traditionally, market leaders present their new flagship mobile devices every year. Many companies release new software together with their new hardware. All together, this sets up a new standard for the whole mobile software industry in terms of user interfaces, screen aspect ratios, navigation tools, and APIs.
A great example of such a drastic change is the release of the iPhone X and iOS 11 with their new safeAreaInput values, gesture interface, revolutionary display shape (aka “screen fringe”), artboard size, pixel density, new typography, and so on. What seemed to be unusual and even unwanted has become a permanent feature of modern mobile hardware. Planning to develop a mobile digital product, you have to balance out the most recent OS version with a few older versions as well. Just don’t let outdated platforms distract you from the one you should target the hardest.
The feud between iOS and Android has been around as long as these operating systems. Technical comparisons aside, mobile developers and testers have to deliver perfectly functioning digital products regardless of the platform they’re built on. As we already know, this highly depends on software testing practices. Are these any different for iOS and Android? If so, how? Here is the list of core factors that set Android and iOS software testing apart.
Unlike iOS, which is exclusively distributed by Apple Inc. and runs only on Apple-branded mobile hardware (iPhone, iPad, etc.) without any customization, Android’s policy regarding customizable OS features is far more democratic. Hardware producers that chose Android as a core for their products are allowed to create custom user interfaces, hiding the core Android mobile architecture beneath design patterns of their choice. The most known custom user interfaces based on Android include One UI by Samsung, EMUI by Huawei, and MIUI by Xiaomi. All of these differ in not only aesthetics but also performance and speed. Understandably, when it comes to Android app testing, QA teams have to spend extra time checking performance and usability on different custom user interfaces, in addition to devices and OS versions.
In terms of codebase accessibility, the two operating systems also demonstrate drastic differences. iOS is a closed-source system based on the XNU kernel. Programming languages prevailing in iOS are Swift, C, C++, and Objective-C. Apple mobile software development standards are quite strict and ensuring the application’s adherence to these standards is amongst the key responsibilities of iOS app testing teams. Android OS, in turn, has an open-source codebase owned by Google with the OS core being mostly based on Linux and written in C and C++. Google’s policy towards software development and Android application testing has always been rather open and welcoming for engineers. This does not mean that Android mobile app testing and development standards are lower, but definitely more lenient with Google Play contributors. In iOS mobile testing, application updates rarely get approved by Apple’s App Review team after the first attempt.
Due to the fact that Apple maintains its mobile operating system strictly unified, the deployment process with iOS apps usually goes a bit faster compared to Android. This is because Apple tries to maintain similarity in optimization and performance for all the iOS versions currently in use. The phase of preparing your iOS app’s build to be uploaded to the App Store still has a lot of steps to follow, but these will be more or less the same for any iOS version. With the operating system by Google, or better to say with every smartphone/tablet model that didn’t receive the latest OS version or runs on a custom UI, Android app testing services have much more work to do.
Application updating is a very important part of mobile software development and testing. Apple’s App Store update review and approval process are a lot longer than on Google’s Play Market. Waiting for your app’s update to be approved might be annoying, but it does not mean there are no pros to scrupulous update reviewing. Even though Android users receive application updates faster, people who prefer iOS gadgets are less likely to witness their favorite apps crashing because of a poorly tested update build. Both mobile operating systems require extra attention from the software testers when it comes to updating, but specifically with Play Market, nobody wastes too much time checking if your particular update is worth releasing.
Numbers to consider:
Obviously, mobile testing cannot be discussed separately from software development. Just a few years ago, testing used to be viewed as yet another phase of a digital project taking its place as follows:
Research and conceptualization
This Waterfall methodology of “develop first, test later” has discredited itself by being inefficient and wasteful for many IT teams. Now, it is being gradually replaced by a holistic, Agile, approach to software quality assurance ― a concept that testing should start as soon as the team gets to work on the project, far before the actual programming. That way, teams can detect not only poorly functioning code but also high-level errors affecting the whole application, as opposed to one particular feature. Following that concept, here are approximate stages the mobile app development process is divided into and a corresponding testing technique for each of them:
Regardless of the experience and qualification, it is extremely hard to come up with a brand new digital product just out of one’s mind. In reality, working on a new application, developers turn to various tricks to diminish the potential risk of failure. Making a prototype and testing it is one of them.
Basically, a prototype is an early sample or model of something that has limited functionality but gives a clear image of future product look and features. Prototype testing allows teams to assess the usability of a future product, its core functions, and try out the whole concept of their application-to-be. Unlike the ready-made application, a digital prototype has no large codebase behind it, which means it doesn’t take long to create one. In fact, you can easily make one in design programs like Figma or Invision. We strongly recommend the prototype testing technique for usefulness validation and early detection of the conceptual inaccuracies.
MVP stands for Minimum Viable Product and is a software development method that implies a new application is first released with core features only, disregarding all the auxiliary ones. After gathering feedback from the first MVP users, the team can proceed with further development. That way, your application gets to users (or a focus group) faster, without wasting time or resources on polishing it to its full potential.
Unlike prototypes, the MVP application has an actual codebase that later can be used as a foundation for the finalized version. In terms of UI/UX design, the MVP version doesn’t focus on the aesthetics much, but we still think that it is a great opportunity to test the overall style and color scheme you’d like to use for the fully-featured product. Working with MVP testing, QA analysts not only test the shortened version before deploying it, but also follow this up with the analysis of feedback from early adopters.
Software testing that takes place right before the release and during it is what most people imagine when thinking of Quality Assurance. And even though you can already see that there’s more to QA than just functional testing at the deployment stage, it is still a fundamental testing phase.
Functional testing compares functional requirements for the product with what has been developed in reality. It is used alongside many other testing types that we’re also breaking down later in this article. Putting it briefly, functional testing focuses on accessibility, main features, basic usability, and potential errors with ways to resolve them.
Don’t think your job is done as soon as your app hits the App Store. Regular updates of mobile programs are just as important as a successful launch. However, even the slightest interruption in the codebase can result in severe bugs and even program crashes. That’s when regression testing comes in the spotlight.
Mobile regression testing is a process of checking whether the new code works fine with older programming. It is aimed at ensuring the application updates do not affect its stability in any way but only make it better. Compared to desktop testing, regression on mobile applications can be more complex to perform because of multiple technical combinations (app architecture ― native or cross-platform, mobile platform, its version, etc.).
Depending on the subject of testing or a particular period of time it takes place, software testing is categorized into different types, levels, and approaches. To avoid getting lost in this maze of QA-related terms, we came up with this straightforward classification that covers the most common categories, approaches, and techniques in the software testing industry as it is today.
Manual testing means all the program checks are being executed by human quality assurance analysts by hand. It is a classic way of software testing that can never be replaced by automated QA fully. Why? First of all, because as long as we develop applications aimed to be used by people, it’s people who should check their quality. This doesn’t mean we underestimate the power of QA automation, we just believe there’s a perfect execution option for every testing method.
Automated testing means that QA engineers write test scripts that execute tests themselves without human involvement. These scripts are oriented at the expected results which they compare to actually received ones from the program. The only thing left after the script did its job is to analyze the results. That way, testing teams can save time and resources needed for thorough quality control.
Due to the conceptual difference between manual and mobile automation testing, it is easy to conclude that not all the testing activities can be successfully automated. Vice versa, some of them become extremely time-consuming and expensive if performed manually. So what processes should be automated with mobile apps specifically? Here’s our reasoned answer:
As you can see, mobile app automation testing holds a lot of potential for product teams. However, you should keep in mind that non-functional aspects of a mobile application require human intellect and perception to be informative. Things like usability, design, localization, and, of course, beta testing should be performed by real people ― the closest thing to your target audience.
Numbers to consider:
Software testing teams can vary greatly in size, position titles, technologies used on the project, and testing methodologies applied. Regardless of that, the key unit of a quality assurance team has always been and remains a QA engineer (interchangeably called software tester or software testing engineer). This job title is rather broad and just from that word combination, you won’t get much information about one’s qualifications, professional experience, and tech stack. So, in case you ever come across a testing department that employs five software testers, most likely each of them does something different operating different technologies and tools.
The table below contains short descriptions of the most common members of QA teams in mobile software testing:
The question of where to locate a quality assurance team is especially puzzling for mobile software projects. Such projects usually aren’t as large as, for example, long-term development of a large system or some legacy application with a huge codebase. Mobile applications do not require too much resources for continuous maintenance and technical support in the long term, and hiring a full-time testing team in this case is not always efficient.
As a way out, many mobile app owners turn to mobile testing outsourcing. Despite the fact owners of digital products have been outsourcing software development for decades, the concept of distant quality assurance was still considered unusual just a few years ago. As of now, the current state of IT service market allows companies of all sizes, from tiny startups to large enterprises, to receive professional mobile app testing services from any location worldwide.
Starting as a cost-saving strategy, software testing outsourcing quickly proved itself as effective as in-house teams. In fact, out of all digital processes including programming, web design, business analysis, and marketing, quality assurance turned out to be the easiest one to entrust to a third party. This is explained by the fact that outside testing teams deliver more transparent and unbiased results compared to in-house QA departments that were involved in the product creation from day one and unintentionally lean towards it.
Mobile QA outsourcing is a great option for people who are just testing the waters by releasing their very first mobile application. As a beginner, you might think your project can’t afford an in-house QA team, therefore, condemning your future app to lousy testing. In reality, it’s not the case. The project-driven collaboration with an outsourcing company allows product owners to prioritize project resources and stay focused on aspects like marketing and on-site promotion while the quality of the code is being taken care of. Also, organizational tasks including hardware, equipment, worksite rent, and human resources management also belong to the outsourcing company’s responsibilities.
According to Payscale, the average annual salary of a software testing engineer equals $56,927 in the United States, which is calculated from the $39,000-89,000 range. As for the hourly rates, US software testers usually stay between $12-55 per one hour of work. However, depending on the level of expertise and place of employment, QA salaries can easily reach six-figure value. Depending on the years of experience, US testing experts salaries distribute as follows:
North America is rightly considered the most expensive software development and testing market in the world. Let’s take a look at other locations to compare the labor market states across them.
The Western Europe region is a slightly smaller job market for software testers, however, countries like Germany and Ireland are known for their skillful QA engineers. The annual salaries here range from $20,000 to $68,000 per year in Germany, $25,000 to 76,000 annually in France, $31,000 to 90,000 in Netherlands, and $28,000-55,000 in Ireland.
Eastern Europe is far more budget-friendly than the two above-described regions. For example, an average Ukrainian software tester earns $18,000 per year. For senior-level engineers, this number grows up to $31,000 annually. In Poland, QA professionals charge for their services from $20,000/year to $31,000 depending on years of experience and technology stack. In general, Eastern Europe has proved to be a perfect combination of high-quality services with reasonable rates.
The Asian region is known for its low labor costs, which, unfortunately, do not always come with service excellence. Still, the local labor market is truly gigantic: you can find 5.2 million software developers in India alone, which means the number of QA engineers can be counted in millions as well. As for the salaries, Indian software testers earn from $2,600 to $11,000 annually. In Pakistan, an average salary of a QA analyst equals $5,000 in one year. China, Japan, and Singapore remain the most expensive development and QA service providers in Asia. There, you can find software testers earning from $23,000 to $67,000.
According to Statista, in 2019 companies allocated 23% of their IT budgets to quality assurance and testing. This mathes the general recommendation for now ― to spend about 25% of all the project resources on software testing and quality control. Of course, this ratio will vary greatly depending on the stage in your software development life cycle, project scope, technology stack, etc. But using that 25% number as your guideline and having chosen the region, you can easily calculate the approximate funds you’ll need for QA.
The time estimation highly depends on the number of people you’re hiring for quality assurance, the qualification level of each of them, the technical complexity of a project, and its scope. Different people can estimate different amounts of time for the same tasks, so we recommend staying flexible during the estimation phase and prioritizing thorough testing over a faster one. Here are the processes you and your team should include in your action plan for QA with the corresponding number of working hours for each:
Given that the efficient software testing process should be started as soon as programmers get to work, it becomes obvious that as a project module, software testing lasts as long as the actual development. However, this does not mean that QA engineers will work as many hours as software developers. On average, software testing takes about 40% of all the project duration time, it just happens on a piecemeal basis. So, if the estimated project duration is 3 months, which equals 66 working days and 528 labor hours, you can expect the software testing to last about 212 hours which is equivalent to 3,5 weeks of work. Now that we know how many hours we’re planning to spend on testing, it is easy to calculate the financial value of it. Working with, let’s say, a software testing engineer located in the US, 212 hours of work would cost us from $7,208 50 to $10,600. A quality assurance engineer from Ukraine would charge about $2,000-$2,500 for the same amount of work, depending on the expertise level.
Choosing a software testing partner is a decision of crucial importance for business. It requires an in-depth analysis of the IT service market, as well as an examination of the teams you consider hiring. But how to tell the difference between a reliable application testing company and a vendor trying to pass for one? To answer this question, we recommend paying attention to the following factors:
The first thing you should do in that case is to go to the software testing vendor, find a contact form and fill it in. The information you provide there is going to be the starting point of your communication with the company and your dedicated team. Obviously, onboarding procedures differ from one company to another, but in general, you can expect the following steps to take place:
Pilot project. Companies, that are confident about their performance and qualification, offer their clients an option to run a small pilot project before signing a long-term contract. This can be a smaller module extracted from your large project or, let’s say, prototype testing which happens early in the mobile app development projects. Hourly rates for the pilot projects can be lower than during the actual deal or you can even get a certain amount of QA labour hours for free.
Large project analysis and planning. If everything went well with the pilot and you want the company to proceed with full-time testing, after quick legal arrangements the assigned team will start diving into the project requirements, develop corresponding testing strategy, and plan the project course.
Testing implementation. According to the plan made earlier, the team will test your project in the appropriate testing environment and report back the testing results to the client. During this stage, a close communication with the project development team will also take place to collectively find the best debugging ways.
Project outcomes. As a result of collaboration with a quality assurance team, you’ll get your digital product tested, improved, and polished to performance excellence. Also, all history of changes implemented during the QA process should be documented and forwarded to you in a readable form.
Technical support. Software projects, especially mobile applications, cannot be considered permanently finished. To stay relevant, continuous improvement and updating are needed. That’s why reputable software vendors provide their clients with lifetime support and on-demand testing services when the app gets updated.
Mobile testing has the same aims and objectives as any other type of software quality assurance ― to check if an application works as expected. However, the mobility itself, rapid pacing of development standards, and many other issues we discussed in this article make mobile QA fundamentally different from desktop software testing. Although meeting technical requirements is necessary, with mobile application development, the work of a product team does not stop there. The value people see (or don’t see) in a mobile app is what defines whether this app is going to stay on their devices and, likewise, in their lives.
Given that the mobile software market is the most competitive sector in the IT industry with 8.9 million apps currently existing in the world, mobile app owners are in no position to skip the testing phase. If you don’t want to put your mobile app launch at risk, take your time finding a truly professional QA team that will maximize the potential of your program and won’t let you release anything less than fail-proof.
42