Such Quality, Much Assurance: From Classroom to QA

Before becoming a QA Tester, I never fully appreciated software testing. While coding, I wrote my unit tests, made sure they passed, and continued on with my life. When I got the job offer to become a QA at Hootsuite, I took it as an opportunity to learn a piece of the giant world of software testing, and now, knowing how valuable QA testing is, I think any software company without a QA team is missing out.

At Hootsuite, most teams have a dedicated QA person. They do a wide variety of things, from testing tickets, to automating the features, to creating tools for the other QA’s to use to do these things, and so much more. The QA’s here fit into the teams just like developers do. We have tickets that are created specifically for us to automate important features, and we participate in estimating how much effort is required for tickets.

You Mean to Tell Me You Can LEARN Things as a QA?

On the iOS team, we test every JIRA ticket that goes through every sprint at least once. Testing a ticket involves coming up with test cases, testing the cases on devices, and eventually automating them. Most of the time we automate a feature before it is released, but sometimes we only automate after being released. Once a feature is completely done, we go through and test each case again to ensure everything runs smoothly.

Test Cases, Test Cases Everywhere

At the bare minimum, a test case needs to cover a common or critical use path. More test cases are required to also cover edge cases and integration. Edge cases are less common use paths while integration tests are to make sure the feature interacts correctly with other features in the app. These are all important to test because we want to ensure a user can’t get our product into an unusable state. Key testing in these areas could be overlooked if we didn’t know our product inside and out. Missing any one of these cases could cause large bugs to reach users, which is never good for anyone. Repeatedly testing key features increases our knowledge of the app, which allows us to do risk analyses to figure out which features could be at risk with any new changes. Being able to pick out these features saves us from having to run a full run through of the app, usually referred to as regression, for every ticket, which is where automation comes in.

Most of our test cases are automated, but those that can’t are tested manually.

 

If time allowed, I would test every case on every device we support. Sadly, ain’t nobody got time for that – but fortunately, it’s really rare for a critical bug to occur on a specific iOS version and device. We save time by testing various Apple products on our supported versions of iOS 8 and iOS 9. If a feature has specific UI elements, we test it on devices with different screen sizes to ensure everything looks the way our designers intended it to. If we find anything that is legitimately wrong, we take the time to iron out detailed reproduction steps to make it as easy as possible on developers to find the problem. Sometimes things can be extremely hard to reproduce, so this process can take a really long time, or sometimes reproduction steps don’t work 100% of the time which can be extremely frustrating.

We catch a lot of bugs 😀

Automate All the Things

Since there aren’t enough hours in a day to test every feature manually on every device and run regressions on the rest of the application, we automate core features. At Hootsuite, the developers push new code several times a day which causes our scripts to require daily maintenance. We have to update our scripts to match any changes made to the application or we report any bugs they catch. All the hassle of updating various scripts is worth it because it saves so much time since we don’t have run full manual regression on every ticket. Overnight we run the scripts on several different devices to ensure we catch any bugs that were created during the day, while covering features on more devices than just the basic four we usually test on.

The framework we use for automation on the mobile team is called Calabash. Calabash supports gestures such as swiping and rotating, which are necessary for some iOS and Android tests. You can also record custom gestures like pinching, to test any fancier features in the app. If our iOS and Android apps were identical, we could even run Calabash for them with the same scripts. Calabash also has a feature that allows you to run commands before adding them to your scripts or inspect UI elements through the command line, which is unbelievably helpful when trying to find a specific label or type. Since our developers add labels to almost every UI element, we can be more exact with our scripts. This also helps make our app more accessibility friendly.

The App Store rejects any app with the Calabash framework embedded in them. We used to have two separate application targets, which is basically like having two separate apps altogether. One was to submit to the App Store with and one to run automation tests with Calabash. With this came a lot of duplication, all the configurations between the two apps were the same, and all the links had to be the same which the developers were required maintain. Testing on a separate build can definitely lead to missed bugs, but we would manually test features on the non-Calabash builds while scripting on the Calabash application to minimize the possibility of missing major bugs. We recently revamped our system such that we just have a Calabash configuration that runs through our main app target, but is only added when we build it with this scheme. This way, we have only one app target, but the App Store won’t reject the app submission since Calabash won’t be in it. This cut in half the links we needed and made it so the developers didn’t have to keep them the same between two targets. We removed the Calabash target completely, and with that the configurations that were copied to it from the main target. We only had to add one configuration for Calabash to run.

Diagram of old Calabash set up with two identical targets with identical configurations, and Calabash only testing on the Debug configuration on one target.

New setup with only one target with an added configuration for Calabash to run.

Calabash runs with a another framework called Cucumber, which executes a language-type of grammar rules called Gherkin. Gherkin has three parts: scenarios, steps, and features.

• Scenarios are individual tests, such as “Try sending a message using a Twitter account”. Scenarios are usually written in plain text style for readability.
• Steps are written in Ruby, and help define what should occur in each scenario. Steps can include other scenario names to execute, or things like scrolling or tapping in the app interface.
• Features are several scenarios and steps together to form a feature file designed to test one major part of the app. For example, one of our features is “Twitter” which has scenarios for all the tests we want to run with Twitter like sending messages from secured or limited accounts or retweeting tweets.

An example of a Scenario which tests if resetting the message cache works for Facebook streams.

A step definition of scrolling down to a stream and tapping it.

Daily Alphas and Betas on Mobile? Oh My!

We use Jenkins to control and monitor our automation on several automation computers. We make the computers pull any new commits that occur every 30 minutes or so from both our scripting repository and our developer repository, running only if there are new commits to grab. Currently, we have specific tests to run on each computer, but we are working on having a master/slave configuration, which will allow the master to send out each test to any available computer. This helps us move from continuous integration towards continuous delivery, and will allow us to submit alpha and beta builds on a close to daily basis by creating a build on master whenever all the tests pass on the slaves. Being able to submit alpha and beta builds daily makes it easier for us to quickly ship out a version of our app to a subset of users, who can then experience new features almost as quickly as they would on the web application. Having this kind of configuration also makes it easier to scale up our architecture, and it is more secure since only the master is allowed to contact the github server.

Picture of our iPhone 6 iOS 9 scripts passing on Jenkins. Always fun to see first thing in the morning!

QA For the Win

Imagine having to do all these tasks and cover the in-depth test cases, while still being a full-time developer. Yikes. Having a QA on the team offers a separate pair of eyes on the product whose job is to ensure the final product works as intended, which we try to do to the best of our ability.

The belief that QAs can’t be technical is completely false. We can be anywhere from 100% technical by automating full time or developing tools to improve automation or 100% manual by just running through test cases on devices. The ability to be versatile in this sense is one of my favourite parts about being a QA. I get the chance to stretch some of my technical skills while still seeing how the product works from more of a customer’s point of view. We love interacting with the product in ways a developer wouldn’t, which gives us a unique perspective that allows us to come up with even more test cases. A QA has so much to offer to a team, so do yourself a favour, hire a QA for your team today. 😀

About the Author

Dana is a QA Co-op on the iOS team. She believes all dogs are puppies. She is passionate about learning mobile development and can often be found wasting enjoying her life on video games.