Select Page

Behind the Scenes: Testing of Conversational AI at Testbirds

May 21, 2019 | Webinars, Testing Reality, Usability & User Experience, Usability

These days at Testbirds, we’re talking a lot about chatbots, machine learning, and conversational agents. While these topics have always interested us, a lot of the excitement is due to the fact that a few weeks ago, we announced a new partnership with Cognigy, the market leader in conversational AI. Our partnership is already starting to bear fruit – on May 28th, 2019, we’ll be hosting a webinar together with Cognigy focusing on chatbots and conversational AI.

If you are interested in participating, have a look here.

Testing Chatbots

We cannot stress the importance of frequent testing and training of chatbots, especially since not only the technological standards but also what users expect from chatbots are rising. In one of our former blog posts, we discussed how conversational agents work and can be improved in general. Now, we want to take you behind the scenes on a more operational level – what exactly does our work with chatbots testing look like (e.g., how do we test chatbots using the power of crowdtesting).

Customer is Key – Pre-development phase

Sure, chatbots are popular – Gartner even predicts that in two years, 50% of enterprises will spend more on chatbots than on apps – but when integrating conversational AI into a company’s customer facing processes, it is crucial to not only implement it for the sake of it, but to always keep the initial purpose in mind: customer satisfaction.
Does the customer really need a chatbot to get information about the opening hours of a store? Maybe it’s faster to find through a simple search of the site or the internet. At Testbirds, we have over 350,000 real testers worldwide that help us to identify exactly this: real customer needs.
It’s not often the case that we get involved in a client’s chatbot implementation process before its launch, although we think it makes so much sense. The client gets natural feedback from their target customers regarding how and in what situations they would interact with a chatbot, all in a variety and density no product team could ever come up with.

Chatbot-Testing – Beta-phase

Most of the chatbot test projects we do with our clients start during prototyping phase. In this stage, optimization processes are still very flexible, so the feedback can be implemented directly after testing. With crowdtesting, the capabilities of the chatbot can be tested under real conditions by the target customers. Below is a bit of an insight into one of our recent tests:

Test Case
The company of a client designed a chatbot to offer customers an additional channel in which they can inform themselves about several different topics. Our client wanted to test if this bot delivers information correctly and quickly, while behaving as naturally as possible.

Test Set-up
The designated Testbirds Project manager chose a test group of 12 testers from our crowd (matching the target group of the client’s product, of course). The testers then performed 100 unique interactions each with the chatbot, with actions spanning five different categories. The Testers had to document their findings as follows:

  • “Question/Interaction”: what question was asked by the tester
  • “Bot Answer / Reaction”: what was the chatbot’s answer
  • “The bot`s answer was”: the testers chose an appropriate answer from a drop-down list containing satisfying, unsatisfying, wrong, asks to try again, no answer/reaction and other

After performing and documenting 100 unique actions with the bot, the testers also had to answer another five general questions about their experience with the chatbot:

  • Which of the following words best describes your interaction with the bot? (please choose three to six words): human, complicated, intuitive, unpleasant, inviting, frustrating, simple, awkward, pleasant, dismissive, motivating, technical
  • How would you rate the language used by the bot? (1 = very natural; 6 = very unnatural)
  • How would you rate your overall experience with the bot? (1 = very satisfying; 6 = very disappointing)
  • How likely is it that you would use this chatbot in the future? (1 = very likely; 6 = very unlikely)
  • Is there anything you can think of that you especially liked or disliked about the bot?


Normally a test such as this would take two to three days to set-up. After the testing phase, the Testbirds Project Manager compiles all results in a final report and presents findings and recommendations for actions to the client. If all goes well and the test purpose and different testing categories are well prepared by the client, a Chatbot Usability Test can be executed in only one week. What’s better is that the collected questions and interactions provide additional training data for the chatbot.

Chatbot Training – Continuous Testing Process

As stated above, the technological standards of Artificial Intelligence are constantly improving and the variety of different ways to ask for information makes broad language training crucial. At Testbirds, the natural diversity of the crowd offers a linguistic variety that cannot be guaranteed by developers or individual testers and can deliver hundreds of new data sets the client then can implement into their chatbot training process. UX and QA chatbot testing should not be a one-time project and needs to be integrated as a continuous QA-process to ensure that the chatbot performs on the highest possible level and adds real value for the costumer. With an ongoing testing set-up, the testing and training criteria can be modified at each iteration thereby it is possible to cover more and more topics or to expand the tester group. This way, the chatbot gets better trained at every test.

Interested in Learning More?

On the May 28th, 2019 at 17:00 CEST, we will go much more into detail and talk about how to build and optimize UX of conversational agent and chatbot as well as reveal the “10 Dos and Dont’s when building a chatbot”.

Until then, keep your customers happy and your robots too!

Share it if you like it:

Tag Cloud

Other content that might be interesting for you:

How to deal with the fragmentation of mobile devices

Mobile operating systems Today’s mobile landscape is more diverse than ever. In fact, as of 2015, there were over 24,000 Android devices on the...

Testbirds Holiday Apps: Stockholm

Ever fancied a trip to Scandinavia? Stockholm, a popular tourist destination for those looking to discover northern Europe, has a bustling big city...

Want to play buzzword-bingo? Let’s talk about AI and 5G!

AI is obviously one of the key topics for this year’s Mobile World Congress starting next week. In this context, the “transformative effect on...

I’m Hans Becker and This is My Role at Testbirds

My Role at Testbirds is a new series of blog posts that gives insights into the daily life at the office. Every so often, one of our employees will...

It’s in the numbers: get your channel mobile-ready

Every quarter, the Dutch e-commerce organisation Thuiswinkel releases numbers to their market research regarding all things ‘home shopping’. It...

I’m Emily and this is my role at Testbirds

My Role at Testbirds is a series of blog posts that gives insights into the daily life at the office. Every so often, one of our employees will...

Meet our 100,000th Tester: João

What is it like to start as a new bird in our nest? Testbirds recently crossed the 100,000 testers milestone and we invite everybody to celebrate...

Home Office as a Reflection of Company Culture

Two months ago, we decided that we’re all going to work from home until the situation with COVID-19 improves and we feel safe enough to come back to...

Cognigy and Testbirds investigate Chatbots & Conversational AI in upcoming webinar

In today’s world, where artificial intelligence is revolutionising how we approach and execute labour, the introduction of Chatbots and...

We want you! How to find good talent as a start-up

Finding the right people is one of the biggest challenges for start-ups. First, there was only us three founders, we were small and not (yet)...

Testbirds specialises in the testing of software such as apps, websites and Internet of Things applications by using innovative technologies and solutions. Under the slogan, “Testing Reality”, the company offers various testing methods to its clients looking to optimise the user-friendliness and functionality of their digital products. With over 250,000 registered testers located in 193 countries, Testbirds is one of the world’s leading crowdtesting providers. In addition, the IT service provider utilises cloud based technologies to support customers in the optimisation of their digital products. The combination of the two testing methods delivers a unique and extensive portfolio that takes the quality of software to the next level. Testbirds was founded in 2011 by Philipp Benkler, Georg Hansbauer and Markus Steinhauser. Today, the company has over 100 employees. Other than its headquarters in Munich, there are now offices in Amsterdam, London and Stockholm, franchises in Hungary and Russia and sales partners in Italy.

© Testbirds GmbH. All rights reserved.

@ Contact