How and Why Developers Should Code for UI Automation

VMC’s SDET3/DEV 1 Chris Stephenson looks at when and why to use element IDs, the usefulness of events for automation, test content, and other insights to help get your UI automation running smoothly and accurately. 

Yep, it happened: your manager just announced that all projects will now have automated UI testing. Why?! It’s been tried. It never gets us anywhere. After spending hours running the tests, more often than not, they’re broken, which means more hours trying to figure out what broke the tests. When they aren’t broken, they’re reporting “bugs” that aren’t bugs, just out-of-date tests. And then there are theTestAutomationIcon bugs that DO get through the UI automation.

The benefits from automation are well known for things like networking and API calls. What if we could get some of the same benefits in our UI test suite? It can be done, but not without some work. Let’s take a look at the UI automation pain points and see what developers can do to help fix them.

“I’m Going to Need to See Some ID”

UI automation is slow. To some degree, that is inherent in the process. Unlike a unit test or API call test, the UI automation needs to bring up the UI, with all of the underlying objects. This takes time and memory. Once this is done, the tests can actually start performing their assigned actions. In most cases, that’s something along the lines of scanning through all of the elements of the UI under test, then doing a series of comparisons against various properties to find the element that needs to be interacted with. Frequently, that goes something like this:

“Go find the 2nd element that is a container, then look inside that for an element that has ‘Eat at Joes’ in the title.”

So now the automation has to scan all the elements by type looking for containers, then dig into the children of the second container and read the titles of all of those elements until it finds ‘Eat at Joes’. Alternately, the test could do this:

“Go find the element with ID ‘JoesSign1’.”

Now the automation scrubs through just the IDs of all the elements, which are intentionally made readily available for accessibility coding, and returns the element with the matching ID. In most cases, this is a couple orders of magnitude faster.

By simply editing the automation ID to something unique, a great deal of time can be saved. But that’s not all. Let’s look at that first test command again:

“Go find the 2nd element…”

The automation scans through the object model from top to bottom, root to leaf. We’re adding a new feature to the product that is going to put another container-type element between the 1st and 2nd container elements. What just happened to our test that is running that command? Yeah, busted. So again, simply updating an existing field to something unique will go a long way toward speeding up and stabilizing UI automation. If you only do this one thing, and stop reading here, you will have done wonders for your UI automation. But wait, there’s more.

“The Waiting Is the Hardest Part”

I can’t begin to tell you how many tests I’ve refactored, or worse, written, with something like this in the code:

“Press the big red button, then wait for…oh, I don’t know, 30 seconds?.. then check to see what happened.”

Really? You hit the button to make it do something. You know it’s going to do something, that’s why a button was there. Why are you waiting for a set time? Just listen for the something to be done. Oh. Right. There is no OnButtonDone event to listen to. This is the root cause of the “UI automation takes forever to run” complaint. Nothing slows automation down as consistently or as heavily as Thread.Wait();. As a developer, you made that button. It does a thing. You know what that thing is. You know when it’s done. So add a simple OnButtonDone event, and fire it when your button is done doing its thing.

I hear you. “Just reduce the wait time – problem solved”. Sure, 30 seconds seems excessive. Unless your button is going to query a database that frequently has a lot of work to slog through, and the database is set to a 30 second timeout. If I don’t wait at least as long as the timeout, I run the risk of calling the test a failure even though the data came back as valid, if a little slow. But what about the days when the database isn’t busy? When the data returns in 30ms? How much time saved, test over test, by being able to definitively know when it is safe to check the return data? The results are generally painful, all because of a simple OnButtonDone event.

“What, Exactly, Are We Looking at Here?”

Many products are designed to provide users with a way to interact with some kind of dynamic content. As such, the products tend to use the content as a support and driver in UI development and real-time behavior. Unfortunately, automated tests are far more difficult to design in a similarly adaptable manner because they need to be able to definitively say “yes, that worked” or “nope, it’s busted.” Changes in content dramatically increase the complexity, and therefore stability and maintainability, of UI automation.

The best way to get around this particular problem is to provide test content. You already have a mechanism to swap content easily (that’s the point of your product), so use that feature to give the tests a little love. Build out test content and provide an access method to the tests that can be used to replace the live content. Alternately, provide your test engineer with a mechanism for inserting their own content at test run-time.

To be clear, this is for testing around truly dynamic content that changes without alteration of the product code, and can potentially change the way the product code behaves. Anything with a data-driven catalog of “things” that are altered outside of the product and subsequently displayed or interacted with by the product user (any inventory system, ever) would qualify. If the “dynamic content” is feature changes, or a set number of scripted states that can’t be altered without rebuilding the product, then this doesn’t really apply.

“So, When You Say ‘No’, What You Really Mean…”

You’re building an awesome product, and you don’t want an ugly error message detracting from that when some underlying mechanism glitches, so they get filed away in some dusty directory somewhere, assuming that they get filed away at all. Those glitches are what tests are looking for.

I agree that smacking the user with an error isn’t the preferred course, but test automation needs this data. Specifically, what happened, when, and why, in real-time. Looping back to the OnButtonDone concept, throw in a little bit of data for the tests to catch. A simple success/fail bool is a good start. An enum that can be referenced at run-time to determine the “why” of the failure is even better.

If I’m writing a test that opens a UI, pushes a button, then reads the resulting dialog, I’ll be able to file a much cleaner bug if my test tells me “I hit the button, and it failed because the DB is not responding” instead of “I ran the test, and the dialog wasn’t there.” My test doesn’t know about a database. It doesn’t care. It’s just telling me what the button told it. This way, no one needs to spend hours trying to figure out why the dialog wasn’t there – it was called out in the test.

“Are You Talking to Me?”

I certainly hope so. Clear and constant communication is the best defense against UI automation problems. When a developer and a tester work closely together on a product, automated tests turn out faster, more robust, and more accurate. A tester can’t really tune their automation until the product feature is checked in and functional. They can pre-build and mock out feature behavior, however, and will be far closer to accurate with direct guidance from the developer that is working on the feature. This minimizes the gap between feature complete and automation complete.

Constant communication also clears out a lot of the “false positives” test automation tends to be known for. If the automation is expecting the result of the button push to be a confirmation dialog, and that dialog was removed to minimize user clicks, well, the test will fail on what is ultimately a “good” behavior. It may be that “everyone knew” that the feature was changing, but it doesn’t hurt anything, and helps a great deal, if you touch base with the test engineer anyway.

I hope this helps to clear some things up, and perhaps explains why your test engineers are frequently muttering under their breath about IDs, events, and content.

How and Why Developers Should Code for UI Automation

Adapting a QA Lab for Virtual Reality

 

VR is changing the QA process with its unique requirements. We asked David Kilgour, VMC’s Global Platforms Test Manager, to tell us some of the factors VMC took into account when expanding its VR test facility.

Since VR and AR began showing commercial viability, there have been parallel discussions about both the potential and challenges of the technology. Developers have steadily improved the virtual experience, yet factors such as headset prices, motion sickness, and limited application outside of gaming have been obstacles to widespread adoption. Now, as the industry focuses on perfecting the visual and auditory experience, the user base has grown exponentially, and this is driving more industries to explore VR’s potential, including entertainment, education, online shopping, fitness, and eager marketers in every industry.

QA will be a critical factor in the growth of both VR and AR, and while the goals are the same – get the highest-quality product to market as quickly as possible – there are several key differences between VR testing and traditional console, PC, and mobile-based QA. Some are obvious, others are subtle, but they were all important considerations as we expanded our VR test environment.

Space

While traditional QA can be squeezed into nearly any available space, VR testing needs a lot of room. Testers will be moving around with no sense of the physical space around them. What might be simple for a regular tester to notice, a VR tester may not. This extends beyond the configuration of the furniture and into more daily tasks: if IT is working on an issue for the adjacent tester, the extra people and items are easy for traditional testers to notice but become a hazard for VR testers who, even if they know the extra items are there, may lose track when they’re deep in testing. To maintain a safe working environment, everyone has a responsibility to ensure that the environment remains clear during testing.

Sunlight

I’m not saying that all VR testers are vampires – but imagine for a moment they are. In a more traditional environment, window blinds and lighting would be adjusted throughout the day so that testers have sufficient light (artificial and natural) without direct sunlight on their monitors. VR testers can be immersed in their testing environment for hours without any sense of changes to their real environment, so they could remove their headset to face harsh, direct sunlight. This wouldn’t be good for a vampire or for the VR tester. With this in mind, we plan the space so that the team gets the right amount of light without fear of direct sunlight.

Motion sickness

VR motion sickness is a real thing, and because some people experience it more than others, screening testers is important. It doesn’t matter how effective the tester may be, if they get sick after wearing the headset for 30 seconds (our record is three seconds) then they’re not suitable for VR. Even for those who are suitable, most people get a little motion sick after testing for an extended time, so we provide ginger-based items (ginger candy, ginger ale, ginger tea, etc.) because it helps combat VR-induced motion sickness. Testers can also take breaks as they feel they need instead of being tied to a set break pattern, with some testers taking a few minutes every hour while others preferring a longer break after multiple hours of testing.

Documentation (Test Plans, Walkthroughs, Text Files, etc.)

VR testers are immersed within their environment, so any documentation is difficult to reference within the game, especially a complex and time-critical walkthrough document. Steps are easily missed if a test plan has very specific and cumbersome testing path, and scrolling text is difficult to check against a text file. The easiest way to adjust for many of these issues is having someone available for the tester to talk to. It could be as simple as having a second tester read the test plan as the VR tester moves through the virtual word, and having the VR tester read aloud the text they see and having a second tester check it against the text file.

Genre

With traditional QA, the genre of a title is less of a priority than a tester’s experience on a particular console, but in the VR environment, genre needs to be considered when selecting the right team. Consider the scariest horror movie you’ve seen – now turn that into a video game and place yourself in the middle of it with your VR headset. While it might be amusing to have someone on the team scream the first time a ghost or monster appears in front of them, it can become quite distracting if it was to happen all the time. Before we assign a test team, we speak with each tester individually and give them an overview of the game. If the genre isn’t suitable, we find testers who are more interested.

Potential downside

With every new technology, there is always a downside. We joke that with VR, it is difficult to tell the difference between a tester who is concentrating on one particular aspect and one who has simply fallen asleep.

We Grow as VR Grows

VMC’s VR testing facility is very different from the area we started with a year ago, and as VR technology evolves and player expectations change with it, we’ll continue to make improvements to both our VR testing methods and our space.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

Adapting a QA Lab for Virtual Reality

The Path to Proper Mobile Testing

We asked Christian Norton, VMC’s Mobile Manager, to talk about what to look for in a mobile QA partner. 

One of the trends we’re seeing in the mobile space is both new titles and updates being released with performance or launch issues on specific devices. These issues not only leave customers disappointed, they generate negative buzz by getting poor (if not incendiary) user reviews. With the exponential growth of mobile apps and games development, compatibility and thorough update testing are key elements to preventing these situations.IGDA-mobile

When you’re considering a QA partner to test your mobile title, here are several factors you should look at to ensure a great experience on every targeted device:

Experience
What matters here isn’t years in business, but their specific experience: have they tested mobile products similar to yours so they know common pitfalls to avoid? Are they knowledgeable about mobile devices so they can make recommendations on which the game or app should be tested on? Can they identify gaps in a test plan and offer insights on a more effective approach? Do they have in-depth knowledge and experience on mobile compatibility testing, and can they provide manual tests on a wide range of devices? Understanding their expertise in advance enables you to collaborate on a smart, efficient test plan for ensuring your product delivers what it was designed to.

Inventory
Mobile users seem to fall into two camps: those who always want the newest model, and those who refuse to give up their old devices until they stop working. Your testing partner needs to have both the hottest items on the market and older devices that retain a huge user base. Devices are also handed down from parents to children who may be using apps or playing games on older devices or software versions. Whether you are releasing an update or a new title, your QA partner should be able to analyze the latest device trends to determine the best test strategy for your game or app.

Flexibility
It’s great when there’s time to execute a thorough test plan within your project timeline, but what if you find something unforeseen a few days before release, or worse, your new title or update is released and several users complain about problems downloading your app or playing your game? You need a solution fast, and that requires a QA partner who understands the urgency of the situation and can quickly provide an effective test solution to help isolate the problems on your users’ preferred devices.

Whether you’re an indie developer or an established game publisher, QA is more than an outsourced service – it’s an essential element of your product’s success. Look for a QA partner you can count on in any circumstance.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

The Path to Proper Mobile Testing

Five QA Tips for Indies

Marc-Andre Legault, VMC’s Test Manager for games, shares his insights on maximizing the impact of a limited QA budget.

With the rise of affordable technology and a new generation of creators working on games starting in their teens, releasing an indie game is within everyone’s reach. Unfortunately, the side effect of this incredible boom is that most indie titles struggle to stand out and sell in an overly crowded market.

In the days of Bastion, Hotline Miami, and Super Meat Boy, there was a perception that every new indie title was stronger than the last and polished to perfection. Now, with thousands of titles being released, many in a less than optimal state, some see the “indie” tag as synonymous with unpolished and unfinished games.

Impressive graphics and frames per second don’t guarantee a title’s success – it has to deliver an enjoyable experience. Consider the case with Minecraft: part of what made Minecraft an early success was that the ambitious initial release, while nowhere near what it would become, was reliably functional. No matter how good a title’s ideas are, if the functionality isn’t working as intended or the player can’t get through an area without having to deal with severe bugs, it won’t survive in a market where many titles live or die by their initial release. Even recent AAA titles have struggled to come back from their initial release states even after multiple patches were released.

This is where a good QA partner can help, ensuring that an indie release offers a satisfying experience, unencumbered by bugs. Since indie titles are usually self-financed and every dollar matters, how does a small team successfully deliver high quality while still controlling costs? The answer lies in how to work with your QA partner, targeting when and where they use their QA resources.

Here are five factors to consider for improving your QA process and maximizing its value:

  • Start at the right time: don’t start so early that many of the launch features are not implemented, and not so late that some UX concerns cannot be reported and addressed.
  • Strategic staffing: once QA starts, try to keep at least one tester on the project (assuming the project is Single Player only) for the duration for the more granular coverage, and bring in a bigger team on key development milestones.
  • Create a good debug tool: limited QA time means making the most of time available, and having the right debug commands available can make a difference in reaching your goals.
  • Create and maintain a Game Design Document: regular updates and/or detailed build notes reduce the chance of invalid issues being reported. While this is recommended for any size project, it’s crucial for a small team/budget.
  • Keep a portion of your budget for post-release: one side effect of having a smaller QA team is that once the title is deployed to the general public, a higher number of out-of-path issues will be found and will need addressing.

By keeping these factors in mind, an indie team can have a good portion of the QA groundwork laid out. In an industry where indie titles are forced to fight for space in the market, good QA can be a real differentiator.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

Five QA Tips for Indies

The Power of Three

Joe Lyons, VP of Business Development, discusses adding value to meetings by inviting a third party.  

The power of three is a concept I’ve seen in many contexts, and I want to share how I’ve incorporated it into my management and sales process with tangible results.

If you’re unfamiliar with the power of three, it’s an idea built around the fact that people naturally gravitate to groups of three. From the Three Bears to the Three Musketeers, from Aristotle’s three modes of persuasion to Thomas Jefferson’s three inalienable rights, the power of three appears continually in our lives.

I made it a part of my management style when a former director added a third person to our 1-on-1 meetings. Including a third person added value by bringing new perspectives while keeping the group small enough that everyone was engaged. It led to us asking better questions and enabled us to probe into what people were thinking about and working on. Plus, an extra mind keeps people honest about what comes out of the meeting. Everyone is on the same page.

Expanding Your 1-on-1s

The value of 1-on-1 meetings is well documented. But in my experience, adding the third person—let’s call it a 1-on-1-on-1—has noticeably changed the value to my employees. They like the format, and gain more insight and holistic understanding. It’s a big improvement if your 1-on-1s have become too routine or rote. Whether I include someone from a different department, a peer, or another executive, the conversations take on a different dynamic. When there’s a third person involved, it even forces me to think differently about the conversation so I can take advantage of that person’s perspective and expertise.

The Power of Three in Action

When I saw the value of the power of three for my team, I wanted to bring the same results to my meetings with clients, and found that creating interdependencies between three stakeholders can help drive business.

Here’s an example: I was working with a client that had a back-to-school deadline and wanted to launch an initiative in August. Unfortunately, Legal said it would take 12 months for them to sign off on everything. I saw no way to meet the deadline on our own, so I asked the client if there was anyone else we could work with to meet the launch deadline. As it turned out, they had a managed service provider that we had already been working with. Boom, we got the deal done by working a pass-through with the MSP. The addition of the third party—the power of three—allowed the client to close the deal. We all benefited.

Here’s another: I was discussing a deal with a prospect that sells IoT solutions to other companies, providing them a service. Because my service ran on Amazon Web Services, and the device that the customer sells is driven through amazon.com, the power of three popped into my head. Not only did I have a better chance of closing that deal, but everyone was interdependent on each other: The prospective customer is selling its product on Amazon, my service was running on AWS, and Amazon wins on both fronts if they do the deal. Presenting this thinking to Amazon, they bit—hard—saying they’d never seen anyone do a deal this way before. It was this outside-the-box thinking that led to a win – for everyone involved.

A New Approach to Sales?

Businesses—at least the smart ones—understand the importance of interdependencies. You scratch my back, I scratch yours. While you may not be able to group three entities together in all of your deals—in some instances it just won’t make sense—there are likely a few opportunities where a three-company partnership will be beneficial for all parties.

Great salespeople understand the importance of switching things up from time to time and trying new tactics. Roll the dice on the power of three, be patient, and see what happens. Chances are you’ll love the results.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

The Power of Three

VMC’s EXP Lab Helps Indie Developers Grow

Independent developers are among the most creative talents in the industry, surprising audiences with fun and engaging new IPs. Many indies have a strong vision for their game or app, but they lack access to the array of knowledge required to transform their brilliant idea into a successful product.

VMC’s EXP Lab is an innovative new incubation program that gives indie developers access to a secured lab environment and information, technology, and mentoring that will enable them to focus on making great games. This program enables developers to learn from industry experts who have worked on numerous indie, mid-sized, and AAA games, including attending informational sessions aimed to help indies prepare for success.

Along with space and infrastructure in a secured lab environment at our headquarters in Redmond, EXP Lab participants will benefit from:

  • Mentorship in QA, development, and production cycles
  • Technical consultancy to meet first-party requirements on all platforms
  • Financial, Legal, PR, and Marketing expertise
  • Local community in games

Even the most talented games professionals can face game development challenges: minimal experience with thorough QA/localization testing, limited understanding of platform certification requirements, limited budgets and access to hardware, and/or insufficient information on best practices for start-to-finish development. EXP Lab is designed to fill these gaps and provide guidance for addressing a wide variety of business challenges.

VMC’s mission is to ensure the most innovative companies in the world deliver an excellent product experience to every customer, everywhere – and many of the most innovative ideas are coming from indie developers. We have a long history of supporting indies, many of which have become industry leaders, and EXP Lab continues this ongoing effort to give indie developers the tools and knowledge to grow.

VMC’s expertise ensures ​the most innovative companies ​in the world deliver an excellent ​product experience to ​every customer, everywhere. Learn more at vmc.com

VMC’s EXP Lab Helps Indie Developers Grow

The Power of Play – April 26th, 2017

One Day. One Industry. Infinite Fun.
The Power of Play Countdown Begins!

 

Join VMC at THE game industry event in the Seattle area on April 26th! Hear from Bungie, Blizzard, Xbox and more! As a sponsor of this fabulous event, we are providing a discount link for 30% off so you can join us: https://www.eventbrite.com/e/power-of-play-2017-registration-29453983658?discount=ARFGames

From the biggest companies like Xbox, Bungie, Blizzard, and Oculus to the best of the indies like Ark Survival, Oxenfree, and Darkest Dungeon, all will be at Power of Play do some sharing, learning and celebrating.

On Wednesday at the Meydenbauer Center in Bellevue you’ll hear top industry speakers, mingle at the award reception, and see the newest indie and VR games around.

For more information go to: www.powerofplay.us. For more information on VMC or to set up a meeting with us, please email info@vmc.com.

At Power of Play You’ll:

* Learn from some of the biggest names in the business like Xbox, Blizzard, Wizards of the Coast, Oculus just to name a few.

* Hear VR experts discuss the near-term state of the market. What will be the single most under-tapped utility in VR? What is being done to address this? Where has VR delivered above or below expectations?

* Watch the top eight finalists present in front of our industry judges as they compete for the Seattle Indie Game Competition crown and $2,500!

The Power of Play – April 26th, 2017