Select Page

Behind the Scenes: Testing of Conversational AI at Testbirds

by | May 21, 2019 | Webinars, Testing Reality, UX, Usability | 0 comments

These days at Testbirds, we’re talking a lot about chatbots, machine learning, and conversational agents. While these topics have always interested us, a lot of the excitement is due to the fact that a few weeks ago, we announced a new partnership with Cognigy, the market leader in conversational AI. Our partnership is already starting to bear fruit – on May 28th, 2019, we’ll be hosting a webinar together with Cognigy focusing on chatbots and conversational AI.

If you are interested in participating, have a look here.

Testing Chatbots

We cannot stress the importance of frequent testing and training of chatbots, especially since not only the technological standards but also what users expect from chatbots are rising. In one of our former blog posts, we discussed how conversational agents work and can be improved in general. Now, we want to take you behind the scenes on a more operational level – what exactly does our work with chatbots testing look like (e.g., how do we test chatbots using the power of crowdtesting).

Customer is Key – Pre-development phase

Sure, chatbots are popular – Gartner even predicts that in two years, 50% of enterprises will spend more on chatbots than on apps – but when integrating conversational AI into a company’s customer facing processes, it is crucial to not only implement it for the sake of it, but to always keep the initial purpose in mind: customer satisfaction.
Does the customer really need a chatbot to get information about the opening hours of a store? Maybe it’s faster to find through a simple search of the site or the internet. At Testbirds, we have over 350,000 real testers worldwide that help us to identify exactly this: real customer needs.
It’s not often the case that we get involved in a client’s chatbot implementation process before its launch, although we think it makes so much sense. The client gets natural feedback from their target customers regarding how and in what situations they would interact with a chatbot, all in a variety and density no product team could ever come up with.

Chatbot-Testing – Beta-phase

Most of the chatbot test projects we do with our clients start during prototyping phase. In this stage, optimization processes are still very flexible, so the feedback can be implemented directly after testing. With crowdtesting, the capabilities of the chatbot can be tested under real conditions by the target customers. Below is a bit of an insight into one of our recent tests:

Test Case
The company of a client designed a chatbot to offer customers an additional channel in which they can inform themselves about several different topics. Our client wanted to test if this bot delivers information correctly and quickly, while behaving as naturally as possible.

Test Set-up
The designated Testbirds Project manager chose a test group of 12 testers from our crowd (matching the target group of the client’s product, of course). The testers then performed 100 unique interactions each with the chatbot, with actions spanning five different categories. The Testers had to document their findings as follows:

  • “Question/Interaction”: what question was asked by the tester
  • “Bot Answer / Reaction”: what was the chatbot’s answer
  • “The bot`s answer was”: the testers chose an appropriate answer from a drop-down list containing satisfying, unsatisfying, wrong, asks to try again, no answer/reaction and other

After performing and documenting 100 unique actions with the bot, the testers also had to answer another five general questions about their experience with the chatbot:

  • Which of the following words best describes your interaction with the bot? (please choose three to six words): human, complicated, intuitive, unpleasant, inviting, frustrating, simple, awkward, pleasant, dismissive, motivating, technical
  • How would you rate the language used by the bot? (1 = very natural; 6 = very unnatural)
  • How would you rate your overall experience with the bot? (1 = very satisfying; 6 = very disappointing)
  • How likely is it that you would use this chatbot in the future? (1 = very likely; 6 = very unlikely)
  • Is there anything you can think of that you especially liked or disliked about the bot?


Normally a test such as this would take two to three days to set-up. After the testing phase, the Testbirds Project Manager compiles all results in a final report and presents findings and recommendations for actions to the client. If all goes well and the test purpose and different testing categories are well prepared by the client, a Chatbot Usability Test can be executed in only one week. What’s better is that the collected questions and interactions provide additional training data for the chatbot.

Chatbot Training – Continuous Testing Process

As stated above, the technological standards of Artificial Intelligence are constantly improving and the variety of different ways to ask for information makes broad language training crucial. At Testbirds, the natural diversity of the crowd offers a linguistic variety that cannot be guaranteed by developers or individual testers and can deliver hundreds of new data sets the client then can implement into their chatbot training process. UX and QA chatbot testing should not be a one-time project and needs to be integrated as a continuous QA-process to ensure that the chatbot performs on the highest possible level and adds real value for the costumer. With an ongoing testing set-up, the testing and training criteria can be modified at each iteration thereby it is possible to cover more and more topics or to expand the tester group. This way, the chatbot gets better trained at every test.

Interested in Learning More?

On the May 28th, 2019 at 17:00 CEST, we will go much more into detail and talk about how to build and optimize UX of conversational agent and chatbot as well as reveal the “10 Dos and Dont’s when building a chatbot”.

Until then, keep your customers happy and your robots too!

Share it if you like it:

Tag Cloud

Other content that might be interesting for you:

Apple enthusiasts wanted! iPhone 8 & iPhone 8 plus now available to be tested in the Nest

September is always the exciting month of the year for all the Apple fans. Now, lots of them will go crazy today: the new iPhone 8 and iPhone 8 plus are available in stores. Learn more about how to become a tester and use your fresh apple device.

Internet of Things – What it is all about and how you can use the power of the crowd to test it

Wouldn’t it be great if your refrigerator could be aware of all the food you need and mail the grocery list to your smartphone while you are on the...

TestChameleon™ in Motion

Over the last couple of weeks, we have been investigating Testing Reality. There have been a number of informational posts on what Testbirds...

App in den Urlaub: Flug-Apps auf dem Security und Usability Prüfstand

The Testbirds story is a classic startup dream come true€7 million is a lot of money. It is the amount of money we were able to secure as a new...

Virtual reality testing and its issues

Virtual reality is transforming the world around us. From gaming to cinema, virtual reality provides an unprecedented experience by placing the user...

German UPA Webinar: “UX als Erfolgsfaktor”

Testbirds, as one of the members of the German UPA (German Chapter of the Usability Professionals Association) is proud to inform you about an...

Testbirds named a “Cool Vendor” by Gartner

We are thrilled to announce that Testbirds has been recognized as a “Cool Vendor” by Gartner in the field of “Enterprise Mobility” for 2014....

Testbirds Starting the New Year with a Kick-Off

All around the world, the New Year was welcomed with spectacular fireworks. Testbirds, however, welcomed 2014 with a big bang. What is a Bird...

Testbirds Holiday Apps: Amsterdam

Are you ready for a holiday? In our new series Testbirds Holiday Apps our employees from all over the world present the must-have mobile apps that...

Christmas at Testbirds: Four ways of celebrating

With offices in Germany, the UK, the Netherlands and Sweden, Testbirds are a very international team. That is why when it comes to Christmas,...

Testbirds specialises in the testing of software such as apps, websites and Internet of Things applications by using innovative technologies and solutions. Under the slogan, “Testing Reality”, the company offers various testing methods to its clients looking to optimise the user-friendliness and functionality of their digital products. With over 250,000 registered testers located in 193 countries, Testbirds is one of the world’s leading crowdtesting providers. In addition, the IT service provider utilises cloud based technologies to support customers in the optimisation of their digital products. The combination of the two testing methods delivers a unique and extensive portfolio that takes the quality of software to the next level. Testbirds was founded in 2011 by Philipp Benkler, Georg Hansbauer and Markus Steinhauser. Today, the company has over 100 employees. Other than its headquarters in Munich, there are now offices in Amsterdam, London and Stockholm, franchises in Hungary and Russia and sales partners in Italy.

© Testbirds GmbH. All rights reserved.

Subscribe to the Testbirds Whistler!

Receive updates on our innovative testing services, webinars, brand-new Nest features!

You have Successfully Subscribed!

@ Contact