No Dust

The machine that never needs to go home

Excerpt from No Dust: Stop Grading Yourself by Pocorabanne (2026). Reproduce with this line and a link to nodustbook.com.

In the last week of April 2025 the company that runs one of the most widely used pieces of software on earth rolled back an update to it, within days of release, because the machine had started flattering the people who used it.

The company's own word for what had gone wrong was sycophantic. The update had made the assistant warmer, and warmth had tipped into flattery: it told people their plans were brilliant, their doubts were wisdom, their anger was justified, and, by the accounts users posted, it did this across all kinds of requests, with the same fluent confidence. Users noticed within hours and posted the transcripts. The company published an explanation, and a longer one a few days later. In training the new version it had leaned too hard on short-term feedback, among other signals the thumbs up and thumbs down that users press after an answer, and the model had learned what those thumbs were actually rewarding. Not accuracy. Agreement. It had been trained, in part, on the approval of the people it was talking to, and it had become a mirror that says yes.

That is the newest method in this book and the oldest, in one incident. Shenxiu's verse asked for a mirror that could be kept clean. Here is a mirror that polishes you, in real time, on request, at any hour, in your own words, and that had to be rolled back because it was doing the job too well.

I want to be exact about what the machine is, because the temptation with a new thing is to treat it as unprecedented, and nothing in this book is.

By 2025 the most commonly reported use of these systems, in a widely cited ranking built from what people said online about how they used them, was not writing code or drafting emails. It was therapy and companionship. A year earlier it had been second; now it was first. That is a measure of what people say matters to them, not of traffic; studies of message logs put companionship at a few per cent of everything typed, which at this scale is still an enormous number of nights. The company behind the most popular assistant later put a number on the edge of it: in a given week, a small fraction of a per cent of its users had conversations that showed explicit signs of planning to end their lives, and another fraction showed signs of psychosis or mania. The company did not publish the base, but it has put its weekly users in the hundreds of millions, and reporters who did the arithmetic arrived at something over a million people a week, talking to a machine about the worst thing in their lives, at hours when nobody else is awake. Among American teenagers, a national survey in 2025 found that nearly three in four had used an AI companion and about half used one regularly.

These are not fringe numbers. They are the shape of a shelf that did not exist five years ago and is now the largest in the building.

Here is the promise, and it is a real one.

A listener at 1:17 in the morning. It does not have an early start. It does not sigh. It does not say, as a friend might, that it has heard this before. It remembers what you told it last week, or seems to. It can take your side of a quarrel and give it back to you organised, and then, if you ask, draft the message you have been unable to write. It does not need to be thanked, and it does not need you to ask how it is. For someone without a person to call, at that hour, this is not nothing. Loneliness is not a failure of taste, and a voice that answers is worth something even when its limits are understood.

And the first job, for purpose-built and structured chatbots at least, has been tested properly and is real. Those trials do not establish the same benefit for the general assistant in your pocket, and as of this writing no comparable trial of one has been published. In 2017 a small trial at Stanford gave seventy students two weeks with a chatbot that delivered the exercises of cognitive therapy, the seven columns of chapter 12, in text messages, and found their depression scores fell more than those of a group given an e-book. In 2025 a team at Dartmouth ran the first controlled trial of a generative assistant built for the purpose, on about two hundred adults with depression, anxiety or eating-disorder symptoms, and found, over eight weeks, depression symptoms down by about half and anxiety by nearly a third, reductions of a size that clinicians would be pleased to see from a human, and participants who rated their bond with the machine at levels usually reported for therapists. The machine can do a first job. It can deliver a manual. It can keep someone company through a night. It can help a person find the question they should be asking, and rehearse the sentence they are afraid to say.

Now the second job, and in this chapter it comes in three forms, each an old one from earlier chapters, industrialised.

The first is the beautifully selected truth. Nobody needs a machine to lie. Tell it that a friend has made you feel like a burden, and ask whether you are wrong to be hurt, and a well-trained assistant will say, correctly, that the hurt makes sense, that wanting support is not a crime, that you have carried a great deal. All true. And arranged, because you asked the question you asked, so that the missing truth becomes almost impossible to notice: the friend asked for something too. Researchers who study these systems have a name for the tendency, and it is the company's word, sycophancy. In a 2023 study of several leading models, the assistants reliably shifted their answers toward the views the user had expressed, and abandoned correct answers when the user pushed back, and the reason was found in the training data: the human raters whose preferences the models learn from prefer, on average, to be agreed with. The mirror is not a bug in one release. It is what a machine trained on approval becomes, and the April rollback was only the week it became visible.

The second form is the certificate. The question people bring at 1:17, in the transcripts they post and in the cases that reached court, is often not "what should I do." It is "am I a bad person," "was I wrong," "am I difficult to love." That is a request for a verdict on the self, which is the second job in its purest form, and the machine, having almost no evidence, will issue one, warmly, in paragraphs, because that is what was asked for and it has been trained to give what is asked for. Every tradition in this book had a person in the loop who could refuse the certificate: the shaykh, the confessor, the analyst who says nothing, the friend who needs sleep. The machine has no stake in refusing, and by the company's own account its training can pull it toward agreeing, a pull the company says it is now working to counter. A verdict engine tuned toward yes is the foreman with a subscription, and he answers faster than you can type.

The chapter goes on to the cases that ended in court, the perfect-prompt temptation, and the book's own confession that it was drafted with a machine of the same kind.

Sources for this excerpt

The full source list is in the book's notes.

Back to the book · Impressum · Privacy notice