Automation and growing reliance on compuers makes us dumb

By NICHOLAS CARR*

Artificial intelligence has arrived. Today’s computers are discerning and sharp. They can sense the environment, untangle knotty problems, make subtle judgments and learn from experience. They don’t think the way we think—they’re still as mindless as toothpicks—but they can replicate many of our most prized intellectual talents. Dazzled by our brilliant new machines, we’ve been rushing to hand them all sorts of sophisticated jobs that we used to do ourselves.

But our growing reliance on computer automation may be exacting a high price. Worrisome evidence suggests that our own intelligence is withering as we become more dependent on the artificial variety. Rather than lifting us up, smart software seems to be dumbing us down.

It has been a slow process. The first wave of automation rolled through U.S. industry after World War II, when manufacturers began installing electronically controlled equipment in their plants. The new machines made factories more efficient and companies more profitable. They were also heralded as emancipators. By relieving factory hands of routine chores, they would do more than boost productivity. They would elevate laborers, giving them more invigorating jobs and more valuable talents. The new technology would be ennobling.

Then, in the 1950s, a Harvard Business School professor named James Bright went into the field to study automation’s actual effects on a variety of industries, from heavy manufacturing to oil refining to bread baking. Factory conditions, he discovered, were anything but uplifting. More often than not, the new machines were leaving workers with drabber, less demanding jobs. An automated milling machine, for example, didn’t transform the metalworker into a more creative artisan; it turned him into a pusher of buttons.

Bright concluded that the overriding effect of automation was (in the jargon of labor economists) to “de-skill” workers rather than to “up-skill” them. “The lesson should be increasingly clear,” he wrote in 1966. “Highly complex equipment” did not require “skilled operators. The ‘skill’ can be built into the machine.”

We are learning that lesson again today on a much broader scale. As software has become capable of analysis and decision-making, automation has leapt out of the factory and into the white-collar world. Computers are taking over the kinds of knowledge work long considered the preserve of well-educated, well-trained professionals: Pilots rely on computers to fly planes; doctors consult them in diagnosing ailments; architects use them to design buildings. Automation’s new wave is hitting just about everyone.

Computers aren’t taking away all the jobs done by talented people. But computers are changing the way the work gets done. And the evidence is mounting that the same de-skilling effect that ate into the talents of factory workers last century is starting to gnaw away at professional skills, even highly specialized ones. Yesterday’s machine operators are today’s computer operators.
Just look skyward. Since their invention a century ago, autopilots have helped to make air travel safer and more efficient. That happy trend continued with the introduction of computerized “fly-by-wire” jets in the 1970s. But now, aviation experts worry that we’ve gone too far. We have shifted so many cockpit tasks from humans to computers that pilots are losing their edge—and beginning to exhibit what the British aviation researcher Matthew Ebbatson calls “skill fade.”

In 2007, while working on his doctoral thesis at Cranfield University’s School of Engineering, Mr. Ebbatson conducted an experiment with a group of airline pilots. He had them perform a difficult maneuver in a flight simulator—bringing a Boeing jet with a crippled engine in for a landing in rough weather—and measured subtle indicators of their skill, such as the precision with which they maintained the plane’s airspeed.

When he compared the simulator readings with the aviators’ actual flight records, he found a close connection between a pilot’s adroitness at the controls and the amount of time the pilot had recently spent flying planes manually. “Flying skills decay quite rapidly towards the fringes of ‘tolerable’ performance without relatively frequent practice,” Mr. Ebbatson concluded. But computers now handle most flight operations between takeoff and touchdown—so “frequent practice” is exactly what pilots are not getting.

Even a slight decay in manual flying ability can risk tragedy. A rusty pilot is more likely to make a mistake in an emergency. Automation-related pilot errors have been implicated in several recent air disasters, including the 2009 crashes of Continental Flight 3407 in Buffalo and Air France Flight 447 in the Atlantic Ocean, and the botched landing of Asiana Flight 214 in San Francisco in 2013.

Late last year, a report from a Federal Aviation Administration task force on cockpit technology documented a growing link between crashes and an overreliance on automation. Pilots have become “accustomed to watching things happen, and reacting, instead of being proactive,” the panel warned. The FAA is now urging airlines to get pilots to spend more time flying by hand.

As software improves, the people using it become less likely to sharpen their own know-how. Applications that offer lots of prompts and tips are often to blame; simpler, less solicitous programs push people harder to think, act and learn.

Ten years ago, information scientists at Utrecht University in the Netherlands had a group of people carry out complicated analytical and planning tasks using either rudimentary software that provided no assistance or sophisticated software that offered a great deal of aid. The researchers found that the people using the simple software developed better strategies, made fewer mistakes and developed a deeper aptitude for the work. The people using the more advanced software, meanwhile, would often “aimlessly click around” when confronted with a tricky problem. The supposedly helpful software actually short-circuited their thinking and learning.

The philosopher Hubert Dreyfus of the University of California, Berkeley, wrote in 2002 that human expertise develops through “experience in a variety of situations, all seen from the same perspective but requiring different tactical decisions.” In other words, our skills get sharper only through practice, when we use them regularly to overcome different sorts of difficult challenges.

The goal of modern software, by contrast, is to ease our way through such challenges. Arduous, painstaking work is exactly what programmers are most eager to automate—after all, that is where the immediate efficiency gains tend to lie. In other words, a fundamental tension ripples between the interests of the people doing the automation and the interests of the people doing the work.

Nevertheless, automation’s scope continues to widen. With the rise of electronic health records, physicians increasingly rely on software templates to guide them through patient exams. The programs incorporate valuable checklists and alerts, but they also make medicine more routinized and formulaic—and distance doctors from their patients.

In a study conducted in 2007-08 in upstate New York, SUNY Albany professor Timothy Hoff interviewed more than 75 primary-care physicians who had adopted computerized systems. The doctors felt that the software was impoverishing their understanding of patients, diminishing their “ability to make informed decisions around diagnosis and treatment.”

Harvard Medical School professor Beth Lown, in a 2012 journal article written with her student Dayron Rodriquez, warned that when doctors become “screen-driven,” following a computer’s prompts rather than “the patient’s narrative thread,” their thinking can become constricted. In the worst cases, they may miss important diagnostic signals.

The risk isn’t just theoretical. In a recent paper published in the journal Diagnosis, three medical researchers—including Hardeep Singh, director of the health policy, quality and informatics program at the Veterans Administration Medical Center in Houston—examined the misdiagnosis of Thomas Eric Duncan, the first person to die of Ebola in the U.S., at Texas Health Presbyterian Hospital Dallas. They argue that the digital templates used by the hospital’s clinicians to record patient information probably helped to induce a kind of tunnel vision. “These highly constrained tools,” the researchers write, “are optimized for data capture but at the expense of sacrificing their utility for appropriate triage and diagnosis, leading users to miss the forest for the trees.” Medical software, they write, is no “replacement for basic history-taking, examination skills, and critical thinking.”

Even creative trades are increasingly suffering from automation’s de-skilling effects. Computer-aided design has helped architects to construct buildings with unusual shapes and materials, but when computers are brought into the design process too early, they can deaden the aesthetic sensitivity and conceptual insight that come from sketching and model-building.

Working by hand, psychological studies have found, is better for unlocking designers’ originality, expands their working memory and strengthens their tactile sense. A sketchpad is an “intelligence amplifier,” says Nigel Cross, a design professor at the Open University in the U.K.

When software takes over, manual skills wane. In his book “The Thinking Hand,” the Finnish architect Juhani Pallasmaa argues that overreliance on computers makes it harder for designers to appreciate the subtlest, most human qualities of their buildings. “The false precision and apparent finiteness of the computer image” narrow a designer’s perspective, he writes, which can mean technically stunning but emotionally sterile work. As University of Miami architecture professor Jacob Brillhart wrote in a 2011 paper, modern computer systems can translate sets of dimensions into precise 3-D renderings with incredible speed, but they also breed “more banal, lazy, and uneventful designs that are void of intellect, imagination and emotion.”

We do not have to resign ourselves to this situation, however. Automation needn’t remove challenges from our work and diminish our skills. Those losses stem from what ergonomists and other scholars call “technology-centered automation,” a design philosophy that has come to dominate the thinking of programmers and engineers.

When system designers begin a project, they first consider the capabilities of computers, with an eye toward delegating as much of the work as possible to the software. The human operator is assigned whatever is left over, which usually consists of relatively passive chores such as entering data, following templates and monitoring displays.

This philosophy traps people in a vicious cycle of de-skilling. By isolating them from hard work, it dulls their skills and increases the odds that they will make mistakes. When those mistakes happen, designers respond by seeking to further restrict people’s responsibilities—spurring a new round of de-skilling.

Because the prevailing technique “emphasizes the needs of technology over those of humans,” it forces people “into a supporting role, one for which we are most unsuited,” writes the cognitive scientist and design researcher Donald Norman of the University of California, San Diego.

There is an alternative.

In “human-centered automation,” the talents of people take precedence. Systems are designed to keep the human operator in what engineers call “the decision loop”—the continuing process of action, feedback and judgment-making. That keeps workers attentive and engaged and promotes the kind of challenging practice that strengthens skills.

In this model, software plays an essential but secondary role. It takes over routine functions that a human operator has already mastered, issues alerts when unexpected situations arise, provides fresh information that expands the operator’s perspective and counters the biases that often distort human thinking. The technology becomes the expert’s partner, not the expert’s replacement.

Pushing automation in a more humane direction doesn’t require any technical breakthroughs. It requires a shift in priorities and a renewed focus on human strengths and weaknesses.

Airlines, for example, could program cockpit computers to shift control back and forth between computer and pilot during a flight. By keeping the aviator alert and active, that small change could make flying even safer.
In accounting, medicine and other professions, software could be far less intrusive, giving people room to exercise their own judgment before serving up algorithmically derived suggestions.

When it comes to the computerization of knowledge work, writes John Lee of the University of Iowa, “a less-automated approach, which places the automation in the role of critiquing the operator, has met with much more success” than the typical practice of supplanting human judgment with machine calculations. The best decision-support systems provide professionals with “alternative interpretations, hypotheses, or choices.”

Human-centered automation doesn’t constrain progress. Rather, it guides progress onto a more humanistic path, providing an antidote to the all-too-common, misanthropic view that venerates computers and denigrates people.

One of the most exciting examples of the human-focused approach is known as adaptive automation. It employs cutting-edge sensors and interpretive algorithms to monitor people’s physical and mental states, then uses that information to shift tasks and responsibilities between human and computer. When the system senses that an operator is struggling with a difficult procedure, it allocates more tasks to the computer to free the operator of distractions. But when it senses that the operator’s interest is waning, it ratchets up the person’s workload to capture their attention and build their skills.

We are amazed by our computers, and we should be. But we shouldn’t let our enthusiasm lead us to underestimate our own talents. Even the smartest software lacks the common sense, ingenuity and verve of the skilled professional. In cockpits, offices or examination rooms, human experts remain indispensable. Their insight, ingenuity and intuition, honed through hard work and seasoned real-world judgment, can’t be replicated by algorithms or robots.

If we let our own skills fade by relying too much on automation, we are going to render ourselves less capable, less resilient and more subservient to our machines. We will create a world more fit for robots than for us.

* NICHOLAS CARR is the author of “The Shallows: What the Internet Is Doing to Our Brains” and most recently, of “The Glass Cage: Automation and Us.”