Automating QA: The Key to Supercharging Mobile Game LiveOps

Many of today’s top grossing mobile games run on LiveOps - and the most successful LiveOps strategies run on constant testing. 

Experienced game teams understand this, and do their best to implement a culture of testing in their development process. 

However, too often I see game studios - particularly new or relatively new ones - relying on manual testing (which is inefficient) or third party QA platforms (which are expensive) to handle testing.

At a certain stage in a game’s lifecycle, the demand for new content and the rigorous testing needed to support it can’t be achieved solely through manual testing - you have to leverage automation. 

Integrating automation into the testing process for your game’s LiveOps, ideally from the start, is not just the fastest way to grow, it’s really the only way to maintain momentum at scale for top grossing games. 

So how can games actually use automation in the testing process? When should it be leveraged? What tools are there to help?

I’ll answer all of these questions below. Let’s jump in. 

Why manual QA testing isn't enough

They say ignorance is bliss, but in game development, there’s little truth to that statement. A more accurate idiom is “if you didn’t test it, it doesn’t work”. 

Several years ago, I built an app for Apple Watches. It worked well as we built out the product during its development, without any known glitches of significance (emphasis on known). 

The week came for our big release, the culmination of months of hard work and anticipation leading up to our defining moment. Confident that we’d built a stellar product, we were left exasperated when we discovered, as we uploaded the app to Apple’s App Store, that it was too large. Our app exceeded Apple’s size limit. 

We were then forced to scramble, frantically inspecting the app’s size and figuring out how to adjust it in order to fall within the size limit. 

We got there eventually, but it was a highly stressful and costly learning experience. 

Experience since has taught me that what we should have done instead was steadily test the app size by uploading test versions to the store in advance of our publish date.

I’m sharing this story because it’s equally applicable to mobile games and LiveOps today. While it might sound like a rookie error, you’d be surprised by how few game teams properly check their game size as they add new content - just one of countless pieces of the puzzle that need to be scrutinized constantly.

Inexperienced developers tend to fail to recognize that things go wrong all the time, and they need to know about it as and when it happens - rather than realizing too late, like we did. 

The stakes are too high and the downtimes are too expensive for games to risk bugs and other issues when releasing new content. Do not leave testing as an afterthought that you remember to address only at crunch time (i.e. release day). Instead, you need to be relentlessly proactive. And to do this, it’s crucial to make automation your best friend.

Automating your QA: Best practices for testing mobile game liveops

Verifying that every minute detail of a large game functions after any given change is practically impossible. 

What we can do instead is to increase our confidence in a build by strategically testing it in ways that fulfill some requirements:

1. Focus on the critical path. Some bugs are expected and acceptable, but being able to log in, spend money, or use the LiveOps Dashboard should always work no matter what.

2. Provide meaningful & informative results. As all programmers know, a verbal "hey, I found a bug" is no match for a stack trace.

3. Make it economical. A testing method is not useful if everyone prefers to skip it because of the work hours or money the test would cost.

Automated testing is an excellent way to introduce critical path test coverage economically. This way, your game team can focus on testing exciting new game features instead of randomly completing game quests for the five millionth time. If you find yourself testing something over and over again before a release, it's a good sign it's time to automate it!

So what does automated testing actually look like for mobile game LiveOps? Generally speaking, there are two main types of testing: integration testing and performance testing.

Integration testing

Integration testing revolves around ensuring the game works as a whole when you put everything together. Here are four ways to leverage automation in this process:

Use test automation frameworks

In order to make sure your game works as a whole, you need to simulate the numerous ways users might interact with the content. Thankfully, frameworks like Appium or Calabash let you create automated tests that do just that - simulate all the potential user interactions with your new content to validate its functionality. 

Develop test scripts

Write test scripts that cover various scenarios, such as gameplay mechanics, UI elements, network connectivity, and backend interactions. 

Continuous integration 

CI tools like Jenkins or GitHub Actions are useful for integrating automated tests into the development pipeline. This allows them to run automatically whenever you roll out new code - an efficient way to make sure your new content and LiveOps work as intended with existing game features. 

Validate data integrity 

Automate the verification of data consistency between different game components, such as the server, client, and database, to ensure smooth integration.

Performance and load testing

Because there is no way game developers can reliably run daily tests to accurately predict how their code will perform when it hits millions of players, it’s vital to be able to simulate these loads to ensure the code can handle the scale. Who can forget the infamous Diablo failure? The last thing you want to do is become a meme! 

Here are four ways to leverage automation for performance and load testing:

Define performance benchmarks 

First of all, game server KPIs such as CPU usage, memory usage and game action latency (ie. the responsiveness of the servers). In the Metaplay case, our SDK ships with pre-configured Grafana-dashboards for observing the most common game server performance metrics - which makes getting started with this much quicker and easier than it otherwise would be.

Pick a tool to create a heavy load on the game

Use a tool like Metaplay SDK's BotClient to make it look like players are playing the game and utilising all its features: creating new accounts, joining guilds, entering matchmaking, sending chat messages, and so on. We know the leaderboard system works with 100 developers playing, but what about with one million bots? Or a hundred million?

The common mistake here is to have the bots only do a naive set of actions like logging in and upgrading a few units, when the big performance impact is more likely to be in the various multiplayer features. Generating a representative load on the game is key to trustworthy results!

Automate stress testing at scale

Create a way to run the bots in the cloud at scale. For example, the Metaplay SDK's BotClient can be run in Docker containers, making it easy to run on in-house CI systems or massive scale cloud services like AWS Fargate.

With a test pipeline setup, you can integrate it into your other CI pipelines to automatically run the load tests for each QA build on a small volume and maybe manually trigger large test runs before every release. The key here is that running tests is as effortless as possible as a part of your other workflows.

Analyze the results and set up alerts for significant changes 

Now that we have a way to easily run large tests and likely also have the most pressing game performance optimizations out of the way, it’s time to make sure things don’t get worse over time. 

The easiest way is to set up automated alerts in your monitoring system (such as Grafana) to notify your team in case something significantly changes in your chosen KPIs. For example, a 2x increase in server CPU usage would definitely be worth investigating in what was assumed to be a routine content update to the game.

Bonus points for manually checking the game client while a large test is running to make sure that the in-game UI deals well with a very, very large amount of players playing at the same time. It’s not uncommon for things like guild search or leaderboards to need some more optimizations when they suddenly handle a huge player base.

Automated testing = smoother LiveOps

Implementing automation in your LiveOps QA is, at its core, about operational streamlining. 

For games releasing new content frequently, relying on human testing is not nearly enough - there’s simply too much to test and humans tend to get lazy when having to do the simple things over and over and over again. 

Setting up savvy automations that facilitate constant testing of all aspects of your game helps you avoid nasty last minute surprises and unnecessary manual overheads. The only way to keep up momentum with a large team and a live game is to unearth issues as fast as possible so they get solved well in time before the release window. You should spend as little time as possible on the “I wonder if we’ve broken something somewhere?” mindset.

Ultimately, using automation to thoroughly test your LiveOps content lays the foundation for a smoother game experience for your users, less downtime, and in turn more development time spent on improving retention and ARPDAU. 

When it comes to implementing automation in practical terms, there are various tools you can use for different purposes. From continuous integration solutions like Jenkins and Github Actions, to stress testing and monitoring tools like what’s included in Metaplay, there’s enough in the market to ensure you cover all aspects of your game’s QA with automation. 

If you’re still on the fence about QA automation for mobile game LiveOps, or unsure how backend services like Metaplay empower leading developers to leverage it, feel free to get in touch with me via LinkedIn.