This story, as told on Facebook, is either one of the funniest things you’ve ever heard, or one of the worst you could imagine.

Setting up their Roomba vacuum to automatically start cleaning the floor from 1:30 am, the family was blissfully unaware of the horror that would soon unfold.

In short, they have a dog, which used their floor as its toilet after the family had gone to bed. The Roomba glided through the poop and kept running its cleaning cycle throughout the house.

You can imagine the rest.

If that doesn’t say everything about our increasing reliance on smart devices within our homes and what can happen when things go wrong, nothing will.

Yes, smart devices provide unparalleled convenience, but they’re also becoming ever more complex, connected, and relied upon. This means they’re increasingly susceptible to a range of safety, security, and usability issues.

That’s why adequate testing is essential.

Now, while it’s unclear if the makers of Roomba ever tested it against running through a dog’s late-night ‘present’, it couldn’t have been beyond the realms of possibility. Pets do routinely use their owner’s homes as their toilet.

If it hadn’t been tested, it probably should have been.

With that in mind, let’s take a look at some other areas where testing is important.

Bugs, flaws, and updates

In December 2015, the Google Nest thermostat received a firmware update. Pretty straightforward, right? Unfortunately, in the new year, about two weeks after the update, a bug caused the device to drain its batteries and go offline. This meant people were unable to heat their homes or get hot water. For those affected, many had to then follow a nine-point procedure to manually restart the thermostat. Not very convenient for those on holiday and whose homes faced the real possibility of damage from frozen water pipes.

While it’s clear that capturing every bug is cost-and-time-intensive, we’re quickly reaching a point where it’s non-negotiable. Particularly because smart devices are increasingly interconnected through centralized hubs.

When your home’s (or business’) thermostat, security, appliances, and lighting can all be affected by a single flaw, even a minor disruption can cause significant problems. Especially when you increase the stakes.

What might happen once cities start relying more on smart infrastructures? Hospitals? Airports? Car manufacturers? It’s highly likely in the coming years that crashes with these systems won’t just be about lights not coming on at a set time, it could be about putting lives at risk.

When looking for bugs, flaws, and potential security risks, developers must strive for perfection, and testers must insist on truly rigorous testing procedures – at all stages of development, and importantly, during updates.

Security holes

When every smart device can be a potential entry point into a home, a hacked device or router can give criminals access to your data, your movements, and even physical access to your house. Closing as many holes as possible is a clever idea.

Recent research by the McAfee computer security firm discovered vulnerabilities in a connected garage door and also a smart ring that you use to open your front door. They even once found a vulnerability in a connected coffee pot that let them access the home’s Wi-Fi network.

Meanwhile, researchers from the Kaspersky cybersecurity company managed to hack into the smart hub controlling the devices in the home of one of their employees (who was okay with it). Additional research a few years earlier showed that leading hubs contained critical remotely exploitable flaws.

Even way, way, way back in 2013, a Forbes reporter was able to – without hacking experience – gain access to a smart (but not so secure) home in Oregon.

It’s the old story: if someone wants to get in, they’ll find a way. The shocking thing is that today they don’t even need to be in the same country to gain access to your home. But every effort must be made to keep them out.

Sadly, vulnerabilities are even being found in connected medical devices.

The more connected our devices are, the more personal data they contain, and the more they provide ‘convenient’ access to our homes, the more they must be rigorously tested. Otherwise, another online attack, such as 2016’s Mirai IoT botnet might compromise even more smart devices.

Perhaps one day soon artificial intelligence and machine learning will automatically scan for, and potentially, fix cyber threats by themselves? Even then, however, that very intelligence will need to be constantly tested and monitored.

System and power outages

This won’t come as any surprise, but connected devices need power. When a battery loses charge, it can be annoying, but when that device then needs a long time to reconnect to your network, that becomes frustrating. If it can even reconnect.

Especially if you need to manually input passwords and settings.

Pro-tip: Think about installing a battery backup system for your networking devices (that don’t already come with a built-in battery). That way, even if the power goes out, they won’t turn off – so when the power comes back on it should all reconnect without any issues.

What happens then when you’re on vacation? If your home’s power goes out, or the network is interrupted, can your devices reconnect? If the power outage is long enough, will the batteries in your smart thermostat last? Is it even clear why they turned off – or hold on a second! Is someone in your house? Maybe your cameras have been hacked? Did you get any notification of what happened – could they even send a message? Do you have a cellular backup?

If you can’t differentiate between a power or network outage and an actual intrusion, you may end up thinking a natural failure is a hacking attempt, or that an intrusion is just a power or network failure.

When devices need to reset, is it a straightforward process? Or do you need to manually reset (and configure) the device? At what point do people give up on a smart device when they lose all their settings every time the power goes out?

Smart devices need to be developed with these issues in mind, and tested, to ensure they can either stay powered during outages and/or easily connect once the power comes back on.

Network outages

There are also times when devices are affected by other outside issues.

In 2019, an Amazon Web Services data center experienced a blackout, which ruined (fried, would be the technical term) some essential hardware, which was made worse when backup generators failed. This resulted in many smart homes, which used services such as IFTTT, going dark.

Using external servers and the cloud all sounds fantastic until it’s not there. In the end, server issues are always going to happen. It’s not an if, it’s a when and how often. This raises the question: Are your devices able to provide a local backup plan?

This is extremely important when it comes to smart homes that must be connected to the internet. Developers want products that are always on, always sending valuable data, and that are part of an ongoing paid contract. But it doesn’t help the homeowner when there’s simply no network to connect them.

Products need fail-safe components and the ability to be controlled locally when servers are down. Do they alert you when they’re off? Are there manual overrides if you can’t access them via your smartphone? What good is voice control if Alexa is offline? Do they have timers that continue if the network is down?

You may have created the world’s most amazing doorbell that uses motion sensors and WiFi connectivity to ring a chime and send your phone a message when there’s someone at the door, but if the network is down, can the person at the door still press a button?

Importantly, are they tested to ensure that when things do go wrong, they can still function (even if at the bare minimum)?

Learning behaviors

With smart devices increasingly using features that make them more ‘intelligent’ – whether through preference learning, environmental detection, even emotions – they’re also becoming more prone to error.

While it may not be as severe as your Roomba covering your house in excrement, it could be a security camera detecting motion when there is none. Or being unable to distinguish between a dog and the masked man trying to open your door.

Perhaps your smart thermostat gets ‘confused’ because a full household has different preferences for the temperature they like? Such failures can lead to increasingly frustrated users.

It’s also AI’s predictive ability that requires it to store very confidential information, including when you come and go, where you are, even your daily routines and habits. It can recognize your face and know it’s you at the door or entering a room. It has your passwords and credit card details and knows when you purchase something online. What could happen if all of that information fell into the wrong hands (because it definitely has before)?

Because AI is (in simple terms) the ability of the software to follow and mimic human cognition, the testing process needs to be robust and infused with the true human element. Crowdtesting has the ability to give hundreds of people access to the AI’s decision-making processes and see where and how it is getting things wrong (and right).

There’s an old saying that ‘tools are only as good as their users’. With AI and machine learning this may quickly change but it’s important to keep in mind that AI is still fallible. But with many human eyes helping to guide it through development, it can learn from its mistakes, and provide a better experience for its users.

The human element

There used to be a time when one of the biggest mistakes you could make when shopping online was accidentally purchasing more than one of the same items. Today, your smart device can do the ordering for you – even if it wasn’t what you wanted it to do.

As happened to a family in Texas when their six year old daughter “had asked Alexa about cookies and a dollhouse”. Alexa then misinterpreted this as an order. The first time the parents knew about it was when an expensive dollhouse and some four pounds of cookies were delivered to their door.

Apparently, it then got worse when the incident was reported on a local news channel. The reporter said, ‘I love the little girl saying, ‘Alexa ordered me a dollhouse’. This then activated the Amazon Echo’s of those listening to the news and, you guessed it, even more dollhouses were ordered.

Of course, it’s not always a funny story. Connected devices can also be used as weapons in domestic abuse. Smartphones can use apps or have stalkerware installed to track our every movement – without our knowledge. The cameras and/or sensors installed in our doors can now tell someone every time you leave the house. You can connect to a tablet using an app on your smartphone and monitor what someone is doing – as was done in the UK.

Certain rooms can be locked. Question histories on voice-activated devices can be accessed.

Movements can be tracked by where the lighting comes on in the house.

Which can all be controlled by the person who has administrative access.

While developers can’t foresee how their products are used in every case, testing to mitigate ways they can be misused or abused can help build trust in those who use them.

After bad reviews

Ideally, you’ve been doing these tests throughout your solution’s development but sometimes you cut some corners, genuinely thought it was all good, missed a few things… it happens. But what do you do when the bad reviews come flooding in?

The last thing you want to do is take your solution off the market. But this may be one of those times when you need to do just that and put your product through another round of thorough testing.

This is where third-party testers can make a stark difference by not only looking at everything with fresh eyes but by tailoring their tests on the (hopefully) detailed reviews you’ve been getting.

Trusted solutions will always find a home

Whether voice control, home security, remote monitoring, or the hundreds of other possible applications, a fully tested and quality assured product, is what will gain consumer trust. That’s what translates to sales.

For a deeper look into why testing is so essential, click here to read our recent article on the importance of testing.

Share it if you like it:

Tag Cloud

About the author

Maarten is our expert for the manufacturing industry.

Categories

Links

Other content that might be interesting for you: