Software testing outside the box: Inside Justinmind
The Justinmind Development team discuss the importance of software testing, why they do it and how they’re perfecting the process to bring you an even more awesome prototyping platform.
Here at Justinmind, we like to share what we’ve learned as well as our achievements—after all, sharing is caring! We’ve recently written to you about some improvements we’ve made, including how we’ve spiced up our SCRUM technique to make the some of the dull stuff more fun and interactive for the team. Today, we’d like to share our testing methods in the hopes that it might awaken your inner quality ninja!
We take careful testing and quality assurance measures throughout our design process to verify the functionality and usability of our platform, as well as finding and fixing bugs. We test before every release to make sure that we are delivering high quality and value and so that our users are not left with any nasty surprises!
How do we have fun whilst doing this? Read on to find out!
Then and Now
5 years ago, our Development team would perform all tests manually in order to find and fix defects in the software.
“Apart from being tedious, the manual tests were extremely time consuming, limited in scope and didn’t allow us to cover everything.” (Victor Conesa, Project Management VP at Justinmind)
We were only launching a new version twice a year, or so. Many things change over the course of 6 months, and they all have to be tested before the new version can be released. When the platform was more basic, with fewer functionalities, it was somewhat easier to test manually—now it’s impossible. Bernat To, Senior Software Engineer at Justinmind, states that “with the current sophistication of the platform, we’ll never finish testing it manually, there are hundreds of thousands of combinations to try”.
That’s why now, our testing and quality checks are performed at two levels: the manual and the automated. Our automated testing is performed through a series of repeated tests, executed by computers that simulate real user interaction. Fast, reliable and repeatable testing!
And what about the manual testing? Well that’s the spark of genius that makes our way of working unique, innovative and fun!
Download Justinmind today to make awesome game prototypes!
Thinking outside the box
Now, the Development team creates prototypes to test functionality and to understand how the user would interact with the platform once installed. For the engineers, creating prototypes during the manual testing process is extremely motivating as it creates flexibility and variation in their workflow. Of course, work can be fun or drudgery, it totally depends on what you make of it. How engaging do you think prototyping a mock website or app is? …exactly! Well, our team has the answer: gaming prototypes!
When creating and replicating games, the engineers include a variety of interactions and visual elements so that the prototypes really look and feel like the real deal—it’s so much more than child’s play.
“We wanted to make examples that were more interesting. To move away from the boring, static designs that took a long time to create and didn’t really serve a purpose.” (Jordi Paricio, Software Engineer, Justinmind)
Not everything is fun and games, of course. We believe that our method is a smart and effective way of testing an array of features on a broader scale. The team has always tried to make feature-orientated test prototypes, be it a game prototype or a simpler prototype to test individual features. When launching a particular feature, they take into consideration the changes made to the code and try to make use of these in more interesting ways.
The games built using older versions of Justinmind, such as Memory and Noughts & Crosses, are simple and limited. But, as the platform has developed and become more powerful with new features and functionality, such as events, the games have become more elaborate and have been developed to look more realistic. When building games such as Street Fighter, the 15-puzzle and Atari Pong, the team focused on adding rich interactions and events to the prototypes. By building realistic prototypes, they were able to test the quality and performance of more features, as well as debug the platform on more complex levels, before launching new versions.
Bernat says that his favorite Justinmind-created prototype is the Street Fighter design. He recalls that they started building this prototype with a very simple version of Justinmind, to simulate a 2-player fight, where the characters were built with rich text. From there, they moved on to implement new features, including variables and events, and adding animation—slowly building upon the high-fidelity characteristics of the platform that were being developed.
For Bernat, this prototype demonstrated how powerful the platform had become. Now the team could detect errors and performance bugs, which likely would have been missed in building simpler web and app prototypes, which have helped to improve the platform a lot.
“Street Fighter went from being a simple prototype to a simulation of a real game with our platform’s high-fidelity features.” (Bernat, Justinmind)
The way we see it, there are some great advantages to our two step testing process. Jordi explains that “with many tests being done automatically, the team can now take advantage of creating helpful examples for our users”. Apart from the game replicas, the developers create other exemplary prototypes that users are able to download and follow as a guide to build their own, piece by piece.
And, by creating unique and individual prototypes at the testing phase, rather than replicating current websites or apps, the engineers are more likely to find new bugs, and not waste time replicating their colleagues’ work. Our new and improved manual testing process is more a state of mind than it is simply checking requirements. The developers have to think as developers and act as real users.
Testing best practices at Justinmind
Testing is a discipline that we hold in high regard within our design process. We understand the need to test everything, and to test it right—we value our “bug-aware” culture. Ignoring a small bug or design flaw could be a symptom of a larger issue not yet found, and we’re not willing to take that risk. That’s why if we can’t fix a bug, we’ll continue searching until we’ve found a solution, and test again to be 100% confident in our product.
Before we implemented the SCRUM technique, it was quite difficult to adapt to the changes in the market and user demand. Now, with regular project sprints, planning and cross-functional backlogs for the Design, Development and Content teams, we’re continually adapting to the flow of business requirements and the end-user needs.
Our Product Management VP, Victor Conesa, supports the Development team with prioritizing test tasks within each sprint. In terms of prioritizing software requirements across releases, there is always a backlog of tasks and requirements that are ranked in terms of urgency and importance. Victor factors in several criteria when evaluating the priority of the tasks in the backlog—which he does on a daily basis. On one hand, there’s the movement of our competitors, on the other, the feedback from our users and clients—including customer service metrics in our Support portal, the user communities on social media, and our case studies with Justinminders. Apart from this, there’s the unforeseen tasks, maintenance, functionality and bug fixes, usability improvement and the vision of the product to take into account as well.
With his level of expertise and knowledge of what resources we have available to us at Justinmind, Victor’s able to determine where our focus should lie. Our gamified testing technique is proving to be of value in this process, as it allows us to dig deeper and to find and fix bugs in a more effective and engaging way.
Testing and teamwork
Collaboration is a symbol of the new design process. We believe that for any business to thrive, there must be teamwork and the load should be shared. That’s why we consider design and development to be fully integrated and interrelated processes.
Within the Development team, one team member will work on fixing the bugs in the software and then the others will test it to make sure that they’re really fixed. What each team member does within each sprint really depends on the priorities of the Development team as a whole: if you have more time, you test, and so on.
“We are 100% Justinmind!” (Carles Galdó, Software Engineer at Justinmind)
And, as Lead Designer, Sergi Arevalo’s role is also crucial to the testing process. He creates and maintains a design backlog so that whenever the developers are creating new prototypes, they have these designs readily available—not only for new features and widgets, but also for the creative testing phase.
So, what’s brewing on the horizon? Well we’re planning on making some of the tests more automatic, so that the Development team can focus more on building and improving the platform that we know you love! However, we’d also like to make the manual phase even more interesting and engaging. We’re introducing a competition to see who can find the most bugs in the Development team and to calculate the bug-finder ratio better—a Justinmind Games. And a prize will be awarded for the best game replica!
We’re so grateful to our team for providing a continuously efficient and effective testing service! We believe that it’s important to keep teams motivated by keeping the workflow interesting, and sprucing up your software testing process could be a good place to start. So go on, give it a go! You can thank us later 🙂