Skip to content

Who Gets Asked When the Chatbot Is Off?

When a chatbot answers a question you would otherwise ask a coworker, it can spare an interruption but also take away a chance for that person to shape the work.

One ChatGPT user told researchers that during a break from AI assistants, they messaged their mother-in-law about something they would usually ask a chatbot. The researchers don’t say what the message was about. Elsewhere in the study, they mention people asking family members for recipes.

An AI assistant can give you a recipe without involving anyone else. Ask a relative and you might get a different recipe, but you’ve also given them a way into your day. The same choice comes up at work. Getting a chatbot to answer a question can spare a coworker an interruption. It can also mean they never see a decision they could have helped you make.

In a November experiment, researchers randomly assigned 625 regular ChatGPT users in the U.S. to take either 14 days or 1 day away from AI assistants. A browser extension blocked common AI services, and the researchers offered a bonus for sticking to the break. Most participants said productivity was their most frequent use of ChatGPT. What changed wasn’t just how they got things done: the longer-break group reported seeing more people in person and a small drop in loneliness relative to the one-day group in the main analysis.

What a question can start

The researchers didn’t find out how much of that difference came from questions that went to people instead. About 8% of the longer-break group’s written reflections on their social lives explicitly mentioned turning to people for help or interaction they would have sought from AI. Others wrote about being less distracted or paying more attention in conversations they were already having. About 4 in 10 reflections mentioned little or no change.

Not everyone stuck to the break, and about 1 in 5 participants missed the final survey. People in the one-day group were more likely to miss it. A separate analysis of that final survey accounted for the difference in response rates. It pointed the same way as the main results, but for both outcomes the range of plausible values included no difference.

Before the slides are done

Consider a team presentation due today. One person is looking up products that competitors have recently launched. Another is proposing new features. In a separate November study, 248 U.S. workers who already used AI on the job imagined tasks like these. The scenarios described doing the work with or without AI and asked how they would respond to a colleague’s feedback: would they value it, seek it out and use it?

The biggest difference in ratings was for brainstorming. Workers rated their inclination to involve a colleague about 0.4 points lower on a scale from 1 to 5 when they imagined brainstorming with AI instead of on their own. The difference for looking up competitors’ products was smaller and inconclusive. No one had to prepare real slides or send a message. What the ratings suggest is that an AI answer can change the apparent need to ask, and that it doesn’t do so equally for every task.

Sometimes needing less input is the benefit. In a Procter & Gamble workshop, workers on their own with AI and pairs working without it both submitted product ideas that earned higher scores than those from people working alone without AI. And asking the chatbot first need not mean working alone. A developer in a study of software teams described junior colleagues getting answers from ChatGPT, then meeting with the developer or the group to work those suggestions into a solution of their own. In that account, the AI’s answer was something to bring to a conversation, not a reason to skip it.

Back at the presentation, an assistant can give you 5 plausible features. If a coworker knows why the team dropped a similar feature last year, that history won’t turn up just by making the list longer. The useful time to ask why is while the answer can still change what goes on the slide.

Sources