A remote-controlled oven is a safety nightmare

Last month, Monzo published a blog post about how they hire product designers. The blog post got some attention on Twitter over the weekend, and in particular a design exercise for an app-controlled oven:

Following a successful phone call, we’ll give you a small task to see how you respond to a real-life challenge.

The brief is to design a companion app interface for an oven with no physical controls.

A grey rectangle with the words “Monzo Oven” inscribed in faint lettering at the top

It should take about an evening or two to complete, though it’s fine to take as much time as you need to think about the problem. Most people return their tasks to us within 7-14 days.

It’s been criticised from various angles – because it’s a take-home design exercise, because an app-controlled oven is dangerous, because hiring people who will blithely design a bad idea bodes poorly for a bank – but I have an angle I haven’t seen elsewhere.

When I saw this oven, my first thought was: how would a terrible flatmate misuse this oven? This is a topic I’ve written about in the past – even if a product or service is bug-free, a malicious or abusive user could still use it to hurt someone. We should always design with abusive personas in mind, and consider how the things we build might be weaponised.

How could somebody nasty misuse an oven that’s controlled by an app? Before you read on, you might want to try this exercise yourself. How many ways can you think of that an abusive ex or terrible flatmate might use this oven to hurt someone? What sort of things should you consider when designing this oven? I’ll explain some of my ideas after the picture.

Screenshot of a kitchen in a game world. Several units in the kitchen are on fire.
This scene will be familiar to anybody who’s played The Sims.

How many ideas did you think of? How many edge cases and loopholes do we need to cover before it’s safe to ship this oven? I’ve written some of my ideas below, to give you a flavour of the sort of thing I thought about. I’d be interested to hear if you thought of something I didn’t – let me know on Twitter.

Who can control the oven?

The brief doesn’t specify whether this oven is tied to a single person, or if multiple people can control it. Since most households have more than one person, let’s assume it has multi-user controls (although a single-user appliance has plenty of other problems). How do you control who has access?

Access control is extremely hard to do well – both the design of the permissions system, and an interface that means users can understand it and configure it for their needs.

What happens when somebody moves out?

The people in a household change over time: a partner moves in, a flatmate moves out, somebody crashes with you while they’re between places. What happens when somebody leaves?

This is drilling into the complexities of multi-user access control. Any time you have a resource that might be shared by multiple people, you need to think about how you allocate permissions, and how you update or remove permissions over time.

What if somebody malicious can control the oven?

So far I’ve talked about the complexities of access control. Now let’s suppose that access control has gone wrong, and somebody malicious has control of the oven. Maybe you have an unsavoury landlord, or an ex-flatmate you fell out with. How could they make your life miserable?

It’s nice to imagine that all of our users are lovely, wholesome people who’d never hurt anyone. The reality is different: there are bad people in the world, and if you have a sufficiently large user base, some of your users will be bad people. If you don’t think about how they’ll misuse your product or service, you’ll find out when they try it in practice – and other people will get hurt in the process.

Conclusions

This post started as a few tweets, and as I started to write, I thought of more and more ways this could go wrong. I hope you found it useful, and maybe it got you thinking about some failure modes you haven’t considered before.

Most of us aren’t building internet-connected ovens, so the specific ideas aren’t immediately practical. But the general ideas – good access controls, dealing with malicious users, our services being used in unexpected and harmful ways – are broadly applicable.

If you’re building any sort of product or service, abusive personas need to be part of your design process. Safety can’t be an afterthought – it’s easier to head off a problem early, in the design stages, before it gets to users. People can and will use the things we build for evil, and we need to anticipate that in advance.