Software Testing World Cup 2016 Recap: 7 Critical Learnings You May have Missed

Last week, while people were thinking about the weekend, 4 Ourskyers got together to join the Software Testing World Cup 2016 (STWC) Asia Competition,  one of the leading events for the software testing community. Testers from all over the world will team up and join the competition to train up their skills and take home the crown. Roy, Oliver, Ten and I (Joyz) come from different positions in Oursky formed a 1-day team to join this exciting opportunity. I’d like to share with you the things we learned during the 3 hours we had against the rest of the world to test a mobile app.

1) Have a Plan for a World Cup

• The Perfect Pop-up Team

• Joyz and Oliver from Oursky’s QA team are good at reporting bugs and have overall testing basics covered.
• Roy from the Project Management (PM) team is good at finding bugs and has backend expertise.
• Ten from the Sales team is good at writing and give usability feedback.

• What we planned to include in the report:
  • Exploratory test
  • Functional test
  • Integration test
  • Performance test (Joyz)
  • Security test (Roy)
  • Usability test (Ten)

• Have clear roles for the 3-hour active game time
  • Prepare given the original game time schedule sent by organizer (active game time: 5pm-8m)
  • Balance the workload (Joyz, Roy and Ten do bug hunting + one type of test above, and Oliver do bug hunting + reporting)

2) The best tools are the ones everyone uses, not always the industry ‘best’

Since it’s a lean team, we needed to pick tools that everyone could use or learn quickly (including non-developers).

• Official communication tool: Slack (since we are familiar with it form our work at Oursky)
• Official bug report tool: LeanTesting (Oursky uses it it every day)
• Mindmap drawing tool: Mindmup (Many hot keys supported, can quickly draw a complete project mindmap in 15 minutes)
• Performance testing tool: Charles Proxy (The GUI is easy to understand, so even non-developers can use it; captures all HTTP traffic passing through the proxy, so you can know how much and what type of data is transferred while using the app)
• Security / API testing tool: Postman (easy to learn and doesn’t need code, just click to test through the API, mainly testing some GET and POST requests through HTTP)

3) Don’t let misunderstandings take up all your time.

We were originally sent this schedule.

• What actually happened
  • 3pm setup the room, grab all the android devices in Oursky
  • 3:30pm start chatting in the official Slack channel, waiting for the testing app build
  • 4pm received the apk download link
  • We thought the game hasn’t started officially, so we slowly started working on the original plan
    • Understand the app
    • Write the project description and finish the mindmap
    • Arrange the section for each of us to start testing, make sure all sections are covered
    • Try out API test
    • Reporting usability issues including minor ones
    • Discuss on bug severity and read each other’s report
  • 5:30pm while we’re still thinking why there’s no YouTube live for the Asia round, we got this notification:

4) Don’t force 3-hour plans into 1 hour: implement a new solution

We were shocked for a minute. What?!?!

We’d tried to re-confirm the time but the event holder, but we got this instead:

Then Roy, our experienced PM, said, “He said 1hr, so it’s 1hr. Let’s stop arguing with him and change our plan.”

We still wanted to play, and we had 1 hour left. If we kept arguing, we wouldn’t have enough time to do anything. The important thing was to remember the objective: finish a complete report. We quickly went into emergency mode.

5) When you have time, choose the best. When you have no time, choose the fastest.

• So this was our new plan:
  • Cut the performance test
  • the whole team to work on bug hunting
  • Focus on obvious and major bugs.
  • Focusing on one simple, independent task that wasn’t dependent on others is easier to impliment.
  • Ten could write the fastest, so everyone passed the details to Ten to write the reports
  • Our goal was to finish the report. Everyone switched from what they could do the best to what each could do the fastest.

6) Focus on the finish line!

View post on imgur.com

By 7pm, time’s up for bug reporting. We quickly wrapped up the report in the grace period. Everyone was writing in the extra 10-minutes.

Our team fell into sync.

• Roy finished the text for technical sections.
• Oliver finished the text for usability section.
• Ten helped count the bugs numbers and passed to Joyz.
• Joyz finished the graph and pie chart for those numbers
• Report sent on time!

7) Competitions have learnings and takeaways

It was a roller coaster ride! We thought we were so prepared, then we were so stressed for an hour. But, I still had a lot of fun because we got together and somehow managed to finish the impossible. It suddenly makes 3 hours seem like a lot of time!

• Tests are rare outside of school. This competition was a precious chance to “take an exam”. We had to train ourselves to stay focused for a 3-hour sprint.
• Be prepared with a list. We’re glad that we’d written the report template complete enough beforehand, so we could save a lot of time. Understanding which things are important helps you be prepared, reuse materials, and adapt faster.
• Unexpected situations always happen and make up a new plan with 1 focused task. Accept changes quickly and rearranging plans in a short time. Having a good fighting attitude!

If this is your first time hearing about it, I encourage you to join next time!

PS: Check out our new open source BaaS for web, mobile, and IoT services: Skygear.io! It has an iOS, Android and React JS SDK for backend development.