Behind the Scenes: Testing of Conversational AI at Testbirds

by | May 21, 2019 | Webinars, Testing Reality, Usability & User Experience, Usability | 0 comments

These days at Testbirds, we’re talking a lot about chatbots, machine learning, and conversational agents. While these topics have always interested us, a lot of the excitement is due to the fact that a few weeks ago, we announced a new partnership with Cognigy, the market leader in conversational AI. Our partnership is already starting to bear fruit – on May 28th, 2019, we’ll be hosting a webinar together with Cognigy focusing on chatbots and conversational AI.

If you are interested in participating, have a look here.

Testing Chatbots

We cannot stress the importance of frequent testing and training of chatbots, especially since not only the technological standards but also what users expect from chatbots are rising. In one of our former blog posts, we discussed how conversational agents work and can be improved in general. Now, we want to take you behind the scenes on a more operational level – what exactly does our work with chatbots testing look like (e.g., how do we test chatbots using the power of crowdtesting).

Customer is Key – Pre-development phase

Sure, chatbots are popular – Gartner even predicts that in two years, 50% of enterprises will spend more on chatbots than on apps – but when integrating conversational AI into a company’s customer facing processes, it is crucial to not only implement it for the sake of it, but to always keep the initial purpose in mind: customer satisfaction.
Does the customer really need a chatbot to get information about the opening hours of a store? Maybe it’s faster to find through a simple search of the site or the internet. At Testbirds, we have over 350,000 real testers worldwide that help us to identify exactly this: real customer needs.
It’s not often the case that we get involved in a client’s chatbot implementation process before its launch, although we think it makes so much sense. The client gets natural feedback from their target customers regarding how and in what situations they would interact with a chatbot, all in a variety and density no product team could ever come up with.

Chatbot-Testing – Beta-phase

Most of the chatbot test projects we do with our clients start during prototyping phase. In this stage, optimization processes are still very flexible, so the feedback can be implemented directly after testing. With crowdtesting, the capabilities of the chatbot can be tested under real conditions by the target customers. Below is a bit of an insight into one of our recent tests:

Test Case
The company of a client designed a chatbot to offer customers an additional channel in which they can inform themselves about several different topics. Our client wanted to test if this bot delivers information correctly and quickly, while behaving as naturally as possible.

Test Set-up
The designated Testbirds Project manager chose a test group of 12 testers from our crowd (matching the target group of the client’s product, of course). The testers then performed 100 unique interactions each with the chatbot, with actions spanning five different categories. The Testers had to document their findings as follows:

  • “Question/Interaction”: what question was asked by the tester
  • “Bot Answer / Reaction”: what was the chatbot’s answer
  • “The bot`s answer was”: the testers chose an appropriate answer from a drop-down list containing satisfying, unsatisfying, wrong, asks to try again, no answer/reaction and other

After performing and documenting 100 unique actions with the bot, the testers also had to answer another five general questions about their experience with the chatbot:

  • Which of the following words best describes your interaction with the bot? (please choose three to six words): human, complicated, intuitive, unpleasant, inviting, frustrating, simple, awkward, pleasant, dismissive, motivating, technical
  • How would you rate the language used by the bot? (1 = very natural; 6 = very unnatural)
  • How would you rate your overall experience with the bot? (1 = very satisfying; 6 = very disappointing)
  • How likely is it that you would use this chatbot in the future? (1 = very likely; 6 = very unlikely)
  • Is there anything you can think of that you especially liked or disliked about the bot?

Results
chatbots-answers

Normally a test such as this would take two to three days to set-up. After the testing phase, the Testbirds Project Manager compiles all results in a final report and presents findings and recommendations for actions to the client. If all goes well and the test purpose and different testing categories are well prepared by the client, a Chatbot Usability Test can be executed in only one week. What’s better is that the collected questions and interactions provide additional training data for the chatbot.

Chatbot Training – Continuous Testing Process

As stated above, the technological standards of Artificial Intelligence are constantly improving and the variety of different ways to ask for information makes broad language training crucial. At Testbirds, the natural diversity of the crowd offers a linguistic variety that cannot be guaranteed by developers or individual testers and can deliver hundreds of new data sets the client then can implement into their chatbot training process. UX and QA chatbot testing should not be a one-time project and needs to be integrated as a continuous QA-process to ensure that the chatbot performs on the highest possible level and adds real value for the costumer. With an ongoing testing set-up, the testing and training criteria can be modified at each iteration thereby it is possible to cover more and more topics or to expand the tester group. This way, the chatbot gets better trained at every test.

Interested in Learning More?

On the May 28th, 2019 at 17:00 CEST, we will go much more into detail and talk about how to build and optimize UX of conversational agent and chatbot as well as reveal the “10 Dos and Dont’s when building a chatbot”.

Until then, keep your customers happy and your robots too!

Share it if you like it:

Tag Cloud

About the author

Regina Heiß

Regina Heiß

Birds flying high, you know how I feel… whenever Regina is not working in the marketing team for Testbirds, you can find her jamming with her Jazz-Trio.

Categories

Links

Other content that might be interesting for you:

I’m Diego and this is my role at Testbirds

My Role at Testbirds is a series of blog posts that gives insights into the daily life at the office. Every so often, one of our employees will...

Co-innovation: A trending concept that requires testing

After rounding off the latest edition of IFA Berlin, the trends of the past year have come to light and the global reach of the event seems larger...

Webinar: Managing User Experience in a Connected World

IoT continues to transform everyday objects into complex technology, evolving how we interact and experience the world. As more and more companies...

Sports Broadcasting: Trends in the Year of Olympics and EURO 2020

Just a few days ago we witnessed one of the biggest sporting events of the year: the 2020 Super Bowl, a.k.a. Super Bowl LIV. (By the way, congrats...

This is the End of Conversion Rate Optimisation

...at least as you know it.“Conversion rate optimisation (CRO) is a system for increasing the percentage of visitors to a website that convert into...

Are you ready for UEFA EURO 2016? These apps will give you a hand!

App Testing Just a couple of hours to go until the most exciting football tournament of the year kicks off in Paris, France. All around the world...

Will software test itself in the future? How Artificial Intelligence is changing the testing landscape

Whether you are steering your navigation system in your car by voice recognition or telling your smart speaker at home which song to play,...

History’s Biggest Software Fails: Cobalt-60 Disaster

Last time on History’s Biggest Software Fails we looked at The Heartbleed Bug, which has been dubbed “perhaps the worst vulnerability found on the...

Celebrating the Biggest Crowd on Earth

We’re more than happy to celebrate a big milestone in our Testbirds company history - our Crowd recently reached an unbelievable 400,000 testers!...

I’m Christine and this is my role at Testbirds

My Role at Testbirds is a series of blog posts that gives insights into the daily life at the office. Every so often, one of our employees will...

Testbirds specialises in the testing of software such as apps, websites and Internet of Things applications by using innovative technologies and solutions. Under the slogan, “Testing Reality”, the company offers various testing methods to its clients looking to optimise the user-friendliness and functionality of their digital products. With over 250,000 registered testers located in 193 countries, Testbirds is one of the world’s leading crowdtesting providers. In addition, the IT service provider utilises cloud based technologies to support customers in the optimisation of their digital products. The combination of the two testing methods delivers a unique and extensive portfolio that takes the quality of software to the next level. Testbirds was founded in 2011 by Philipp Benkler, Georg Hansbauer and Markus Steinhauser. Today, the company has over 100 employees. Other than its headquarters in Munich, there are now offices in Amsterdam, London and Stockholm, franchises in Hungary and Russia and sales partners in Italy.

© Testbirds GmbH. All rights reserved.

Subscribe to the Testbirds Whistler!

Receive updates on our innovative testing services, webinars, brand-new Nest features!

You have Successfully Subscribed!

@ Contact