Select Page

Behind the Scenes: Testing of Conversational AI at Testbirds

by | May 21, 2019 | Webinars, Testing Reality, Usability & User Experience, Usability | 0 comments

These days at Testbirds, we’re talking a lot about chatbots, machine learning, and conversational agents. While these topics have always interested us, a lot of the excitement is due to the fact that a few weeks ago, we announced a new partnership with Cognigy, the market leader in conversational AI. Our partnership is already starting to bear fruit – on May 28th, 2019, we’ll be hosting a webinar together with Cognigy focusing on chatbots and conversational AI.

If you are interested in participating, have a look here.

Testing Chatbots

We cannot stress the importance of frequent testing and training of chatbots, especially since not only the technological standards but also what users expect from chatbots are rising. In one of our former blog posts, we discussed how conversational agents work and can be improved in general. Now, we want to take you behind the scenes on a more operational level – what exactly does our work with chatbots testing look like (e.g., how do we test chatbots using the power of crowdtesting).

Customer is Key – Pre-development phase

Sure, chatbots are popular – Gartner even predicts that in two years, 50% of enterprises will spend more on chatbots than on apps – but when integrating conversational AI into a company’s customer facing processes, it is crucial to not only implement it for the sake of it, but to always keep the initial purpose in mind: customer satisfaction.
Does the customer really need a chatbot to get information about the opening hours of a store? Maybe it’s faster to find through a simple search of the site or the internet. At Testbirds, we have over 350,000 real testers worldwide that help us to identify exactly this: real customer needs.
It’s not often the case that we get involved in a client’s chatbot implementation process before its launch, although we think it makes so much sense. The client gets natural feedback from their target customers regarding how and in what situations they would interact with a chatbot, all in a variety and density no product team could ever come up with.

Chatbot-Testing – Beta-phase

Most of the chatbot test projects we do with our clients start during prototyping phase. In this stage, optimization processes are still very flexible, so the feedback can be implemented directly after testing. With crowdtesting, the capabilities of the chatbot can be tested under real conditions by the target customers. Below is a bit of an insight into one of our recent tests:

Test Case
The company of a client designed a chatbot to offer customers an additional channel in which they can inform themselves about several different topics. Our client wanted to test if this bot delivers information correctly and quickly, while behaving as naturally as possible.

Test Set-up
The designated Testbirds Project manager chose a test group of 12 testers from our crowd (matching the target group of the client’s product, of course). The testers then performed 100 unique interactions each with the chatbot, with actions spanning five different categories. The Testers had to document their findings as follows:

  • “Question/Interaction”: what question was asked by the tester
  • “Bot Answer / Reaction”: what was the chatbot’s answer
  • “The bot`s answer was”: the testers chose an appropriate answer from a drop-down list containing satisfying, unsatisfying, wrong, asks to try again, no answer/reaction and other

After performing and documenting 100 unique actions with the bot, the testers also had to answer another five general questions about their experience with the chatbot:

  • Which of the following words best describes your interaction with the bot? (please choose three to six words): human, complicated, intuitive, unpleasant, inviting, frustrating, simple, awkward, pleasant, dismissive, motivating, technical
  • How would you rate the language used by the bot? (1 = very natural; 6 = very unnatural)
  • How would you rate your overall experience with the bot? (1 = very satisfying; 6 = very disappointing)
  • How likely is it that you would use this chatbot in the future? (1 = very likely; 6 = very unlikely)
  • Is there anything you can think of that you especially liked or disliked about the bot?


Normally a test such as this would take two to three days to set-up. After the testing phase, the Testbirds Project Manager compiles all results in a final report and presents findings and recommendations for actions to the client. If all goes well and the test purpose and different testing categories are well prepared by the client, a Chatbot Usability Test can be executed in only one week. What’s better is that the collected questions and interactions provide additional training data for the chatbot.

Chatbot Training – Continuous Testing Process

As stated above, the technological standards of Artificial Intelligence are constantly improving and the variety of different ways to ask for information makes broad language training crucial. At Testbirds, the natural diversity of the crowd offers a linguistic variety that cannot be guaranteed by developers or individual testers and can deliver hundreds of new data sets the client then can implement into their chatbot training process. UX and QA chatbot testing should not be a one-time project and needs to be integrated as a continuous QA-process to ensure that the chatbot performs on the highest possible level and adds real value for the costumer. With an ongoing testing set-up, the testing and training criteria can be modified at each iteration thereby it is possible to cover more and more topics or to expand the tester group. This way, the chatbot gets better trained at every test.

Interested in Learning More?

On the May 28th, 2019 at 17:00 CEST, we will go much more into detail and talk about how to build and optimize UX of conversational agent and chatbot as well as reveal the “10 Dos and Dont’s when building a chatbot”.

Until then, keep your customers happy and your robots too!

Share it if you like it:

Tag Cloud

Other content that might be interesting for you:

App Review: World Lens

This week, the Testbirds team have stumbled across a pretty cool augmented reality app called the World Lens App. In this blog post, we’d like to...

Webinar: Managing User Experience in a Connected World

IoT continues to transform everyday objects into complex technology, evolving how we interact and experience the world. As more and more companies...

Coming to dmexco? Win an UX workshop for your digital product!

Let’s talk about UX! Meet us at #dmexco17 for expert insights and a chance to win a UX workshop including €4,000 testing credit! On September 13th...

I’m Ferdi and This is My Role at Testbirds

My Role at Testbirds is a series of blog posts that gives insights into the daily life at the office. Every so often, one of our employees will...

#MWC19 – This is what caught our eyes in Barcelona

After a very hectic time in Barcelona last week, we have finally had some time to reflect on the Mobile World Congress 2019. As always, we traveled...

Meet our Testbirds: Alexis

For this month’s tester story, we have invited Alexis from Belgium to share his Testbirds experience. He is one of the most experienced testers in...

Pre-App Store Rating – What, How and Why?

When developing an app it's often difficult for a developer to truly understand how and what end users want and require to have a great experience....

Top Employer 2018 – How We Live Up To Our Name

Everyone knows that feeling when you receive a lot of praise and you ask yourself ‘What did I do to deserve this?’ Testbirds is “Top Arbeitgeber...

Review on the Caterpillar B 15: Rugged smartphone for digger drivers and outdoor enthusiasts

Have you ever heard about Caterpillar Smartphones? If you are thinking now: “Caterpillar? Don't they produce diggers?” you are completely right. But...

Gastbeitrag: Mein Workflow für Crowdtesting

Der folgende Beitrag kommt von unserem Bird vinni. Wir haben nachgefragt, wie er beim Testen vorgeht und auf was besonders zu achten ist. Ich bin...

Testbirds specialises in the testing of software such as apps, websites and Internet of Things applications by using innovative technologies and solutions. Under the slogan, “Testing Reality”, the company offers various testing methods to its clients looking to optimise the user-friendliness and functionality of their digital products. With over 250,000 registered testers located in 193 countries, Testbirds is one of the world’s leading crowdtesting providers. In addition, the IT service provider utilises cloud based technologies to support customers in the optimisation of their digital products. The combination of the two testing methods delivers a unique and extensive portfolio that takes the quality of software to the next level. Testbirds was founded in 2011 by Philipp Benkler, Georg Hansbauer and Markus Steinhauser. Today, the company has over 100 employees. Other than its headquarters in Munich, there are now offices in Amsterdam, London and Stockholm, franchises in Hungary and Russia and sales partners in Italy.

© Testbirds GmbH. All rights reserved.

Subscribe to the Testbirds Whistler!

Receive updates on our innovative testing services, webinars, brand-new Nest features!

You have Successfully Subscribed!

@ Contact