Two Trends Impacting the Modern Games QA Team

We asked Zack Hiltz, VMC’s Games QA Services Manager, what the future of Games QA will look like. Here are two trends he’s seeing that will affect the composition of future QA teams. 

Throughout my career in Games QA, I’ve been asked questions like, “Where do you see QA being in five years? What will a Games QA team look like then?” I always did my best to provide an insightful response, knowing full-well that Games is an industry like no other, an endlessly variable playground of design, innovation, and emergent technology. No other major entertainment medium presents the same multitude of means by which consumers access and interact with content. It truly is an industry of infinite possibilities, and with it, infinite challenges.ControllerPic2

Despite this, it’s QA’s role in the development lifecycle to prepare and account for every possible outcome to the best of their abilities. When preparing for this role, it is imperative that you start at the beginning, thinking carefully about the type of team you’re going to build and how it will shift and evolve as the production does. To help with this, here are a couple of trends affecting game QA today and points to consider when putting together your next QA team:

The Blurring Line between DevQA and FQA

I’ve worked with QA teams of vastly varying sizes, from quick in-and-out five-person strike teams to massive 200+ tester operations running year-round. Regardless of size, the same principle always applies: no matter how big your QA team, those man-hours of coverage throughout production will never compare to the endless ways thousands of end-users will tear into your code once it goes live. This is further compounded by the increasing complexity of games software year after year.

To combat this, many QA Managers are recognizing the potential futility of simply ramping headcount ever-upward and are instead actively exploring options to maximize their current QA team’s effectiveness. As a result, black-box functional testers are often being exposed to more advanced testing techniques, combining aspects of white-box methodologies to further their understanding of the game’s internal infrastructure without needing to dive as deep into the code. FQA testers are then able to execute more highly-targeted tests, leveraging their advancing familiarity with the game and its debug commands to explore a broader realm of defect detection. For example, working with parameters to further stress engine stability, or comb through data to identify design issues before a black-box tester might uncover it naturally.

This gray-box hybrid approach is already familiar to many who have worked in development or served as DevQA, yet even in a large-scale QA operation, DevQA headcount tends to be incredibly small (proportionately-speaking) due to the nature of being shoulder-to-shoulder with the development team. Nonetheless, studios are seeing a notable benefit in deploying this more code-relevant testing approach to the wider QA team, naturally leading to an ever-shrinking gap between the direct, high-volume quality verification nature of standard FQA and the more nuanced, isolated expertise of DevQA.

Staffing your QA Team with Your Audience

The video game market today is more inclusive and expansive than ever before. It is no longer in the hands of a select few or dominated by any one particular genre; new developers arrive on the scene every day and new genres are constantly being created or revived from ages past.

The reason is audience. The ever-expanding footprint of the games industry across global cultures has led to an incredibly diverse consumer-base where everything is relevant to someone. If you build it, they will come. Conversely, no single game can dominate an industry this big because no single game can be everything to everyone, and some developers are taking note. Rather than casting the widest-possible net, many are aspiring not for global domination, but to instead carve out their own little corner of the industry and double-down on their key player-bases.

But despite an ever-growing population of gamers year-after-year, one thing remains the same:  gamers can be hard to please. In an increasingly eclectic industry, it is imperative that the people in your little corner are satisfied.

To help with this, QA teams are increasingly staffing their projects with the right individuals for the job, not just in terms of technical expertise, but in terms of high-level game expertise, too. Equipping your QA roster with genre familiarity is nothing new, but in the modern gaming market, meeting the increasing quality demands necessitates staffing beyond simple familiarity and doing your best to source expert-level individuals. These Subject Matter Experts (SME) help guide and execute the testing from a perspective that is indicative of both a game’s general target audience as well as the highly passionate subset of fans that tend to be the most vocally active in the community. SMEs can prepare for and catch the hardest-to-find issues because they themselves are among those passionate fans.

This practice is common in franchise development, for example bringing in expert-level fans of an original entry of an IP to test the sequel. But one area where we’re seeing an increase in this practice is the live gaming environment. As online games grow longer-and-longer legs post-release due to unending content cycles and the revenue models to sustain them, the player base in turn becomes increasingly accustomed to the game, its mechanics, and its possibilities. In order to retain these customers, developers need to not only satisfy a constant stream of updates, but also maintain a certain bar of quality that speaks directly to them.  To help accommodate this, QA teams working on live games tend to be staffed at a higher ratio of SME to non-SME than other projects. This ensures that any new patches or content updates are exposed to and filtered through as much subject matter expertise as possible within a much smaller development cycle.

This approach has proven invaluable in the modern games space, and I expect the demand to grow as developers continue breaking new ground, building new experiences, and seeking new audiences.

VMC’s games expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com/games

Two Trends Impacting the Modern Games QA Team

Adapting a QA Lab for Virtual Reality

Embed from Getty Images

 

VR is changing the QA process with its unique requirements. We asked David Kilgour, VMC’s Global Platforms Test Manager, to tell us some of the factors VMC took into account when expanding its VR test facility.

Since VR and AR began showing commercial viability, there have been parallel discussions about both the potential and challenges of the technology. Developers have steadily improved the virtual experience, yet factors such as headset prices, motion sickness, and limited application outside of gaming have been obstacles to widespread adoption. Now, as the industry focuses on perfecting the visual and auditory experience, the user base has grown exponentially, and this is driving more industries to explore VR’s potential, including entertainment, education, online shopping, fitness, and eager marketers in every industry.

QA will be a critical factor in the growth of both VR and AR, and while the goals are the same – get the highest-quality product to market as quickly as possible – there are several key differences between VR testing and traditional console, PC, and mobile-based QA. Some are obvious, others are subtle, but they were all important considerations as we expanded our VR test environment.

Space

While traditional QA can be squeezed into nearly any available space, VR testing needs a lot of room. Testers will be moving around with no sense of the physical space around them. What might be simple for a regular tester to notice, a VR tester may not. This extends beyond the configuration of the furniture and into more daily tasks: if IT is working on an issue for the adjacent tester, the extra people and items are easy for traditional testers to notice but become a hazard for VR testers who, even if they know the extra items are there, may lose track when they’re deep in testing. To maintain a safe working environment, everyone has a responsibility to ensure that the environment remains clear during testing.

Sunlight

I’m not saying that all VR testers are vampires – but imagine for a moment they are. In a more traditional environment, window blinds and lighting would be adjusted throughout the day so that testers have sufficient light (artificial and natural) without direct sunlight on their monitors. VR testers can be immersed in their testing environment for hours without any sense of changes to their real environment, so they could remove their headset to face harsh, direct sunlight. This wouldn’t be good for a vampire or for the VR tester. With this in mind, we plan the space so that the team gets the right amount of light without fear of direct sunlight.

Motion sickness

VR motion sickness is a real thing, and because some people experience it more than others, screening testers is important. It doesn’t matter how effective the tester may be, if they get sick after wearing the headset for 30 seconds (our record is three seconds) then they’re not suitable for VR. Even for those who are suitable, most people get a little motion sick after testing for an extended time, so we provide ginger-based items (ginger candy, ginger ale, ginger tea, etc.) because it helps combat VR-induced motion sickness. Testers can also take breaks as they feel they need instead of being tied to a set break pattern, with some testers taking a few minutes every hour while others preferring a longer break after multiple hours of testing.

Documentation (Test Plans, Walkthroughs, Text Files, etc.)

VR testers are immersed within their environment, so any documentation is difficult to reference within the game, especially a complex and time-critical walkthrough document. Steps are easily missed if a test plan has very specific and cumbersome testing path, and scrolling text is difficult to check against a text file. The easiest way to adjust for many of these issues is having someone available for the tester to talk to. It could be as simple as having a second tester read the test plan as the VR tester moves through the virtual word, and having the VR tester read aloud the text they see and having a second tester check it against the text file.

Genre

With traditional QA, the genre of a title is less of a priority than a tester’s experience on a particular console, but in the VR environment, genre needs to be considered when selecting the right team. Consider the scariest horror movie you’ve seen – now turn that into a video game and place yourself in the middle of it with your VR headset. While it might be amusing to have someone on the team scream the first time a ghost or monster appears in front of them, it can become quite distracting if it was to happen all the time. Before we assign a test team, we speak with each tester individually and give them an overview of the game. If the genre isn’t suitable, we find testers who are more interested.

Potential downside

With every new technology, there is always a downside. We joke that with VR, it is difficult to tell the difference between a tester who is concentrating on one particular aspect and one who has simply fallen asleep.

We Grow as VR Grows

VMC’s VR testing facility is very different from the area we started with a year ago, and as VR technology evolves and player expectations change with it, we’ll continue to make improvements to both our VR testing methods and our space.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

Adapting a QA Lab for Virtual Reality

The Path to Proper Mobile Testing

We asked Christian Norton, VMC’s Mobile Manager, to talk about what to look for in a mobile QA partner. 

One of the trends we’re seeing in the mobile space is both new titles and updates being released with performance or launch issues on specific devices. These issues not only leave customers disappointed, they generate negative buzz by getting poor (if not incendiary) user reviews. With the exponential growth of mobile apps and games development, compatibility and thorough update testing are key elements to preventing these situations.IGDA-mobile

When you’re considering a QA partner to test your mobile title, here are several factors you should look at to ensure a great experience on every targeted device:

Experience
What matters here isn’t years in business, but their specific experience: have they tested mobile products similar to yours so they know common pitfalls to avoid? Are they knowledgeable about mobile devices so they can make recommendations on which the game or app should be tested on? Can they identify gaps in a test plan and offer insights on a more effective approach? Do they have in-depth knowledge and experience on mobile compatibility testing, and can they provide manual tests on a wide range of devices? Understanding their expertise in advance enables you to collaborate on a smart, efficient test plan for ensuring your product delivers what it was designed to.

Inventory
Mobile users seem to fall into two camps: those who always want the newest model, and those who refuse to give up their old devices until they stop working. Your testing partner needs to have both the hottest items on the market and older devices that retain a huge user base. Devices are also handed down from parents to children who may be using apps or playing games on older devices or software versions. Whether you are releasing an update or a new title, your QA partner should be able to analyze the latest device trends to determine the best test strategy for your game or app.

Flexibility
It’s great when there’s time to execute a thorough test plan within your project timeline, but what if you find something unforeseen a few days before release, or worse, your new title or update is released and several users complain about problems downloading your app or playing your game? You need a solution fast, and that requires a QA partner who understands the urgency of the situation and can quickly provide an effective test solution to help isolate the problems on your users’ preferred devices.

Whether you’re an indie developer or an established game publisher, QA is more than an outsourced service – it’s an essential element of your product’s success. Look for a QA partner you can count on in any circumstance.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

The Path to Proper Mobile Testing

Five QA Tips for Indies

Marc-Andre Legault, VMC’s Test Manager for games, shares his insights on maximizing the impact of a limited QA budget.

With the rise of affordable technology and a new generation of creators working on games starting in their teens, releasing an indie game is within everyone’s reach. Unfortunately, the side effect of this incredible boom is that most indie titles struggle to stand out and sell in an overly crowded market.

In the days of Bastion, Hotline Miami, and Super Meat Boy, there was a perception that every new indie title was stronger than the last and polished to perfection. Now, with thousands of titles being released, many in a less than optimal state, some see the “indie” tag as synonymous with unpolished and unfinished games.

Impressive graphics and frames per second don’t guarantee a title’s success – it has to deliver an enjoyable experience. Consider the case with Minecraft: part of what made Minecraft an early success was that the ambitious initial release, while nowhere near what it would become, was reliably functional. No matter how good a title’s ideas are, if the functionality isn’t working as intended or the player can’t get through an area without having to deal with severe bugs, it won’t survive in a market where many titles live or die by their initial release. Even recent AAA titles have struggled to come back from their initial release states even after multiple patches were released.

This is where a good QA partner can help, ensuring that an indie release offers a satisfying experience, unencumbered by bugs. Since indie titles are usually self-financed and every dollar matters, how does a small team successfully deliver high quality while still controlling costs? The answer lies in how to work with your QA partner, targeting when and where they use their QA resources.

Here are five factors to consider for improving your QA process and maximizing its value:

  • Start at the right time: don’t start so early that many of the launch features are not implemented, and not so late that some UX concerns cannot be reported and addressed.
  • Strategic staffing: once QA starts, try to keep at least one tester on the project (assuming the project is Single Player only) for the duration for the more granular coverage, and bring in a bigger team on key development milestones.
  • Create a good debug tool: limited QA time means making the most of time available, and having the right debug commands available can make a difference in reaching your goals.
  • Create and maintain a Game Design Document: regular updates and/or detailed build notes reduce the chance of invalid issues being reported. While this is recommended for any size project, it’s crucial for a small team/budget.
  • Keep a portion of your budget for post-release: one side effect of having a smaller QA team is that once the title is deployed to the general public, a higher number of out-of-path issues will be found and will need addressing.

By keeping these factors in mind, an indie team can have a good portion of the QA groundwork laid out. In an industry where indie titles are forced to fight for space in the market, good QA can be a real differentiator.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

Five QA Tips for Indies

Is It Time to Integrate Your QA and Support?

Kirstin Whittle (Sr. Manager, Business Development) discusses how parallel QA and support services improve products and user experiences. 

Every Developer and Publisher manages their QA and Customer Support differently, and often separately, because QA can be seen as the opposite of Support – QA is finding bugs during development so the consumer won’t see them and Support is the consumer finding bugs and bringing them to the attention of the developer. But in reality, they’re both part of the same process: delivering a great user experience. Aggressive sprint cycles for regular updates and new downloadable content mean work on successful titles never stops, and that makes having efficient processes for both QA and Support more essential than ever – and the hybrid model is emerging as the most effective method of ensuring quality within tight time frames.

What is the hybrid model?

I define it as the bundling of QA and Support by a single provider with connected teams. This process allows your title to have Testing, Localisation and Support managed simultaneously utilising a common knowledge base for economies of scale.

Consider the complexity of the knowledge transfer process with multiple service companies: first, you brief your QA Partner about the project; then you brief your Translation Partner about the project; when that’s complete, you brief your Audio Partner and then the LQA Partner next until you’re finally briefing your Support Partner about all of the issues that were identified in the process. Each partner has a limited understanding of the previous partners’ work, with no simple way to cross-reference issues or ask questions of the other providers. These circumstances not only slow your development process, they increase your management time and cost, reduce accuracy and consistency, and increase the possibility of duplication of process and effort.

The hybrid model streamlines this process by having every team operating from the same knowledge base and easily communicating in real-time about discovered or known issues. The hybrid process accelerates each area through improved knowledge retention and collaboration and gives Developers and Publishers a single point of contact for inquiry and discussion on any aspect of the QA and Support process.

Moreover, the hybrid model delivers a better experience for your users because the Support Team has access to the latest real-time information about bugs, patches, updates, and other known issues. This gives the Support Agent more authority and gives the user confidence that you’re actively addressing issues on the game they want to play. Combined, this ensures an overall better impression of your Company, IP, and Support.

This method can deliver even further economies of scale by deploying Support Teams who are also trained in QA techniques, smoothing out the peaks and valleys of the cyclical development process of QA needs rising pre-launch and user support needs rising post-launch.

Time to market is a critical factor for regularly updated titles, so having a reliable process for effective and efficient Testing and Support is critical to your ability to satisfy and maintain your player base. The hybrid QA/Support model delivers the product quality and customer satisfaction you need on the development schedule your title is committed to. In other words, it’s not just faster – it’s better.

Is It Time to Integrate Your QA and Support?

New Devices Drive New Approaches to QA

Kirstin Whittle (Sr. Manager, Business Development) explains how multi-platform evolution is driving new QA protocols. 

We all have so many devices now – phones, tablets, desktops, consoles, even smartwatches and VR. It means there are many ways for users to engage with games and applications. This variety allows developers to attract and build a strong user base across multiple devices, but it is also driving changes to the role of QA in the product development process.

We all know that people expect a seamless playing experience. They want to remain immersed in the game world no matter what platform they’re playing on. As the proliferation of options grows – more devices, on more platforms, with continual hardware upgrades – the complexity of the QA and production processes across multiple devices grows with it. These changes are creating the opportunity for stronger partnerships between developers and their QA partners.

More Devices, More Demand

Pundits like to predict how the emergence of one technology will be the death knell for another, but the reality is more complex.

While manufacturers and the media evangelise the latest consoles and devices, most users aren’t upgrading with every new release. People get comfortable with their preferred interface or can’t afford to upgrade as often as the manufacturers might prefer. Along with a range of platforms and devices, the QA process must contend with different generations of each.

With so many active platforms, ensuring compatibility (and backwards compatibility) for a multi-platform title requires access to a comprehensive inventory of relevant devices.

Yet for most companies, the limited frequency of need – initial launch and periodic updates – makes assembling a huge inventory of devices a very cost-prohibitive process.

Working with a reliable QA outsourcing firm that owns and maintains a full range of the latest (and not-so-latest) devices allows developers to have their products fully tested on every version of their target operating system, device, and platform.

Continuity Across Every Platform

You learn a lot while performing QA on a title – and that knowledge is crucial to the multi-platform QA and support process. Tight coordination between test teams on different platforms enables efficient cross-referencing of issues across different devices, in turn accelerating results and giving development teams more time to address bugs. This is especially valuable when deadlines are driven by release dates and marketing plans rather than development.

Here’s the bottom line: Players want a great user experience, and delivering consistent product quality across everything from smartphones and tablets to desktops and consoles requires looking at the multi-platform QA process holistically, not as individual QA projects that are specific to each device.

Rather than running the risk of end-user disappointment by limiting the scope of your QA process, it is far better to choose a partner who can expertly handle every aspect of your multi-platform release strategy, including testing and localisation across a full range of devices, console compliance, and whatever else you need.

A well-executed QA strategy is essential to maximising both the efficiency of the process and the quality of the results.

New Devices Drive New Approaches to QA

Why indie developers need to think about more than development

Kirstin Whittle (Sr. Manager, Business Development) shares her thoughts on why creativity alone isn’t enough to ensure success. 

I had the pleasure of attending the 2015 Nordic Game Conference in Malmö and was impressed with so many creative, ambitious new games.

Indie games continue to thrive in Scandinavia and Europe, and I enjoyed talking with many start-up developers along with a number of industry veterans working on new ventures, to learn about what they’re doing and where they want their games to go.

The indie arena is great for developers who want more freedom and control to create the games they want to play, and the ongoing evolution of publishing models makes it possible to develop and release games that likely never would be released by an established publisher that expects a certain return on their investment.

But as is the case for any game, creativity isn’t enough to spur a game’s success.

Prepping for success

When the primary focus is on development, global production support services may not be a key focus, but developers need to have excellence in every area of their business, including marketing, legal, QA, localisation, live game operations and community management.

Managing quality in all of these areas is essential to getting people to play and keep playing your game because a great idea won’t make a splash if it’s poorly tested or localised, or isn’t marketed to your target audience.

No one knows what the next big thing is going to be, and many indie developers aren’t prepared to scale up quickly when their game takes off. Global production support services need to be an integral part of a development plan, but there’s no one-size-fits-all solution for any game.

Here are a few key factors to consider:

Pick a partner, not a provider

A good partner will serve as a guardian of your IP, and will understand that your game is your baby. Many global production support companies will take on indie projects and even offer advice prior to a formal engagement because they also want to be a part of the next big thing.

Talk to support partners who you trust and who have expertise to fill the gaps in the areas your in-house team is lacking.

Get them involved early

Getting your support partners involved early enables developers to benefit
from the insights an experienced partner can provide.

This includes anticipating obstacles and pitfalls, planning for all contingencies, and having additional support specialists who are already in the loop, who know your game, and with whom you have an established rapport.

Think beyond the launch

You want to offer more than just a great game – you want to give your customers a great experience, and this may include live game operations (games-as-a-service), customer support, and various forms of community management.

While an indie game may have humble beginnings, having it become a success doesn’t mean your relationship with your global production support partners has to change.

Many developers see augmenting their staff to bring everything in-house as a sign of success, but the peaks and valleys of the production process mean the workload for different groups will ebb and flow.

Continuing to work with outsourced partners can remain the most cost-effective approach, especially partners who already know your game, have proven their value, and can continue to provide critical subject matter expertise while you maintain complete creative control of your IP.

So pick good partners and get them involved early and you’ll be ready for anything.

Why indie developers need to think about more than development