Our GDC 2024 presentation "How not to f**k your studio: Forecasting done right" now available. Download here >
Let’s take a look at the “Testing is Context-Dependant” Testing Principle, which has become extremely important throughout my career. There are a lot of layers to this principle and I’ve found that it can be applied to more than just your application under test.
I call the following the Context Aspects:
So, you’re building a mobile game app. There are several questions you might ask about it:
These are very important questions to ask to get context about the product. This will help you determine your testing strategy and tooling needs.
Let’s imagine that this a casual game that needs to be easily accessible and engaging. You don’t want players having to navigate multiple menus and screens just to start playing the game. It should be as easy as opening the app, and tapping ‘play’ to start.
You need to test with your user behaviour in mind. Understanding the primary paths players will take in your game (from start, to pause, to in-game mechanics and even closing the game) plays a vital role in building out your smoke tests.
Understanding how the application functions is extremely important. You will need to understand how the game’s backend and frontend integrate. You will need to understand how the data flows through the application and where it gets stored. You will also need to understand how the system gets monitored.
Even though you might not be contributing to the code base, take the time to understand what languages and tools are being used. Take the time to read through the pull requests for the work you are about to test. You can even use that as an opportunity to ask questions. The more you understand, the better you will test.
This one is interesting and certainly a lot more applicable since remote work became a thing. Are you part of a team or working with multiple teams? Are your team members early birds or night owls? Do they work in a different time zone? These might not seem important questions but hear me out.
I once worked in a team where one of the developers started work at 3 am my local time. By the time we had standup the next morning, all the bugs I had logged the previous night would be fixed and ready for testing. This had an amazing impact on our process. Bugs were fixed at an astounding rate and I could validate them while the context was still fresh, allowing us to release faster.
I’ve also had to work with multiple teams at a time. If I had two projects at once, I would do a 50/50 split, or if one of the projects was close to a big release I would adjust the percentage accordingly. Sometimes I was able to give my team context as to why I wasn’t super responsive during certain times of the day and sometimes I wasn’t. It was certainly a lot easier working with the teams that had the context.
This principle of context is also applicable to your testing process.
I was recently told about the concept of YAGNI, which stands for “You Ain’t Gonna Need It”. It’s a principle from Extreme Programming that states a developer should not add functionality until deemed necessary. I found this concept quite interesting so I did some more reading and decided to see how I could apply it to my testing process.
I noticed that a lot of my process was built around traceability and reporting. With my current process, I could tell you the following:
I was super proud of this process. It worked beautifully. I had built an entire test management solution in Notion, which I tweaked as I worked on features. There was only one problem. Almost 80% of my process wasn’t adding any value.
If bugs were found in Production I was never asked for my test report; it wasn’t needed. A bug was found, and it needed to be fixed. In all honesty, I would never go back and review my test report either. I wrote the test cases and executed them so I knew whether the bug was something I had covered or not.
This is where context can be applied. If I was leading a team of QA Engineers or working in a highly regulated fintech industry, the process would be different. I would need to keep a record of what was tested, and when, for compliance reasons. My need was different.
So what did I need?
These steps were part of my original process, they were just bloated and attached to an unnecessary need to report on quality.
Our testing processes exist due to the context of what is required. We can use the concept of YAGNI to implement processes that add value now.
This doesn’t mean that we should throw all of our processes away and do the bare minimum. Take a moment and:
