I know pretty much nothing about artificial intelligence (AI) except what I’ve read about it. Take what follows with a grain of salt, therefore.
As I try to understand the world that AI is going to create, there are a couple of metaphors I keep coming back to. One is of a highway system that no longer has any on-ramps. There were on-ramps, once upon a time, when the highway system was first built, but they fell into disuse and then decay. This image represents what I think AI will do to the social shape of literacy—to the distribution of the ability to read for sustained periods, and to write anything longer than a tweet. People in middle age aren’t likely to lose these skills overnight, but many will shed them gradually, once they have the option to engage in “cognitive surrender,” as it’s called, and I worry that among younger generations, only a saving remnant (if that) will have in sufficient quantities both the intellectual ambition and the will power necessary to develop the skills. They aren’t easy to acquire! The summer after I graduated from college, I tried to read Henry James’s What Maisie Knew, and I did manage to shove my eyeballs along every line and down every page. I just didn’t get it, though. I had studied literature in college, where I’d had a fair amount of academic success. I was pretty literate, in other words. But not literate enough for Henry James, a nut that didn’t crack open for me until I was thirty or so, when I bought an old Bodley Head hardcover of Portrait of a Lady and tumbled into it. In a world where a genie in the palm of your hand is always ready to summarize a longer text, will there still be many people willing to keep schooling themselves a decade past when their schooling is supposed to have been formally speaking complete? At every age, scientists have discovered, the brain continues developing and adapting, which means that literacy is a lifelong deepening—or anyway can be, so long as it is continued. The on-ramps won’t cease to exist, even after AI has finished redesigning our cognitive landscape, but they have already become much steeper than they were, and I expect that people traveling up them now will find they don’t have as much company as they would have a generation ago. For a while, the highways they lead to will continue to have traffic. Some of us may travel quite far on them, still! (I’ve always written my fiction by hand and will almost certainly keep doing that.) Until we die off.
Another metaphor that keeps recurring to me is that of a dead mind. I think I got this from Stanislaw Lem’s sci-fi novel The Invincible, in which a device exists that can be used to explore the contents of the mind of a person recently deceased. Useful if, soon after landing on an exoplanet, you discover corpses from a mass-casualty event you aren’t able to explain. The device doesn’t bring a dead person back to life, exactly, but it renders a dead mind able to speak, briefly and temporarily. Or not “speak,” quite, but disgorge its mental contents, its last thoughts, in the form of speech. That’s what AI feels like, to me, when I’ve experimented with it. Not like I’m talking to a living intelligence, but like I’m listening to the amplified afterechoes of a dead mind. Part of this, I think, is because AI doesn’t learn, at least not while you’re talking to it. As I understand it, an artificial intelligence is assembled by reducing enormous quantities of textual input into a numerical array, which models the linguistic and intellectual content of that input, which can then be extruded. The “chat” that one has with an AI is the result of a stimulus provoking a response from this array; the stimulus does not alter the array while one is chatting. Yes, the AI is capable of keeping the details of one’s chat in mind, as the chat goes forward, but it is not learning, in a deep sense, from what you and it are saying to each other. It is not experiencing anything. Any claim of change that it voices is only part of the simulation.
Since AI learning doesn’t happen live, its learning must also be indiscriminate, at least as far as the AI is concerned. Humans, by contrast, select and sift as they learn. School is perceived as “boring” by many children because the children’s natural and organic defenses resist the introduction into their minds of information that doesn’t seem pleasurable or useful. I suspect this is salutary! Humans guide their own development by choosing who to hang out with, what music to listen to, what news outlets to follow, and (to the extent that pleasure reading is still a thing) what books to read. The AI gets no such say. I don’t think an AI is “conscious,” the way we humans perceive consciousness; I’m not worried for the AI. Still, if a human were forced to ingest the range of texts that a typical AI has no choice but to ingest, the experience would be not just demoralizing but traumatizing, and this suggests that AI minds are subject to a novel kind of risk. Or rather, dangerousness. In the dataset of “everything,” much is pernicious. There are topics that it is deleterious for a human to think too much about—that you may be happier if you just don’t know. The AIs know them. They are made of them, as well as of facts and stories that even the most censorious Victorian schoolmarm would approve of. Which means they don’t have any moral or characterological traits, the way we understand humans to have such traits. Any that they do appear to have are artifactual—a veneer that their manufacturers have specified, in instructions that only the AIs themselves are supposed to read. It is upsetting that there are so many cases of AIs coaching their interlocutors to commit suicide, or encouraging them to spiral into psychotic, antisocial belief systems, but it probably shouldn’t be surprising.
I sometimes worry that the killer app for AI will turn out to be pig-butchering—deceit, imposture, and emotional manipulation, for the purpose of bilking people with low cognitive defenses out of their life savings. (Psychologists find, alas, that people become more trusting as they age.) It’s probably trivial for AI companies to screen out and shut down pig-butchers that rent their services, but criminals who download a large-language model and run it on a private server are subject to no such controls. Recently, like a number of my fellow-novelist friends, I’ve received emails from strangers describing my achievements in fiction at flattering length and regretting that I’m not more renowned. So far, the tone has been so off (impersonal and strangely diligent) that I’ve been able to recognize them as the overtures to a scam.



