Microsoft Word Kurzweil, Ray The Singularity Is Near doc

Strong AI.
Strong AI promises to continue the exponential gains of human civilization. (As I discussed earlier, I 
include the nonbiological intelligence derived from our human civilization as still human.) But the dangers it presents 
are also profound precisely because of its amplification of intelligence. Intelligence is inherently impossible to control, 
so the various strategies that have been devised to control nanotechnology (for example, the "broadcast architecture" 
described below) won't work for strong AI. There have been discussions and proposals to guide AI development 
toward what Eliezer Yudkowsky calls "friendly AI"
30
(see the section "Protection from 'Unfriendly' Strong AI," p. 
420). These are useful for discussion, but it is infeasible today to devise strategies that will absolutely ensure that 
future AI embodies human ethics and values. 
Returning to the Past?
In his essay and presentations Bill Joy eloquently describes the plagues of centuries past and 
how new self-replicating technologies, such as mutant bioengineered pathogens and nanobots run amok, may bring 
back long-forgotten pestilence. Joy acknowledges that technological advances, such as antibiotics and improved 
sanitation, have freed us from the prevalence of such plagues, and such constructive applications, therefore, need to 
continue. Suffering in the world continues and demands our steadfast attention. Should we tell the millions of people 
afflicted with cancer and other devastating conditions that we are canceling the development of all bioengineered 
treatments because there is a risk that these same technologies may someday be used for malevolent purposes? Having 
posed this rhetorical question, I realize that there is a movement to do exactly that, but most people would agree that 
such broad-based relinquishment is not the answer. 
The continued opportunity to alleviate human distress is one key motivation for continuing technological 
advancement. Also compelling are the already apparent economic gains that will continue to hasten in the decades 
ahead. The ongoing acceleration of many intertwined technologies produces roads paved with gold. (I use the plural 
here because technology is clearly not a single path.) In a competitive environment it is an economic imperative to go 
down these roads. Relinquishing technological advancement would be economic suicide for individuals, companies, 
and nations. 

---

The Idea of Relinquishment 
The major advances in civilization all but wreck the civilizations in which they occur. 
—A
LFRED 
N
ORTH 
W
HITEHEAD
This brings us to the issue of relinquishment, which is the most controversial recommendation by relinquishment 
advocates such as Bill McKibben. I do feel that relinquishment at the right level is part of a responsible and 
constructive response to the genuine perils that we will face in the future. The issue, however, is exactly this: at what 
level 
are
we to relinquish technology? 
Ted Kaczynski, who became known to the world as the Unabomber, would have us renounce all of it.
31
This is 
neither desirable nor feasible, and the futility of such a position is only underscored by the senselessness of 
Kaczynski's deplorable tactics. 
Other voices, less reckless than Kaczynski's, are nonetheless likewise arguing for broad-based relinquishment of 
technology. McKibben takes the position that we already have sufficient technology and that further progress should 
end. In his latest book, 
Enough: Staying Human in an Engineered Age
, he metaphorically compares technology to 
beer: "One beer is good, two beers may be better; eight beers, you're almost certainly going to regret."
32
That metaphor 
misses the point and ignores the extensive suffering that remains in the human world that we can alleviate through 
sustained scientific advance. 
Although new technologies, like anything else, may be used to excess at times, their promise is not just a matter of 
adding a fourth cell phone or doubling the number of unwanted e-mails. Rather, it means perfecting the technologies 
to conquer cancer and other devastating diseases, creating ubiquitous wealth to overcome poverty, cleaning up the 
environment from the effects of the first industrial revolution (an objective articulated by McKibben), and overcoming 
many other age-old problems.