Microsoft Word Kurzweil, Ray The Singularity Is Near doc

Protection from "Unfriendly" Strong AI.
Even as effective a mechanism as the broadcast architecture, however, 
won't serve as protection against abuses of strong AI. The barriers provided by the broadcast architecture rely on the 
lack of intelligence in nanoengineered entities. By definition, however, intelligent entities have the cleverness to easily 
overcome such barriers. 
Eliezer Yudkowsky has extensively analyzed paradigms, architectures, and ethical rules that may help assure that 
once strong AI has the means of accessing and modifying its own design it remains friendly to biological humanity 
and supportive of its values. Given that self-improving strong AI cannot be recalled, Yudkowsky points out that we 
need to "get it right the first time," and that its initial design must have "zero nonrecoverable errors."
45

---

Inherently there will be no absolute protection against strong AI. Although the argument is subtle I believe that 
maintaining an open free-market system for incremental scientific and technological progress, in which each step is 
subject to market acceptance, will provide the most constructive environment for technology to embody widespread 
human values. As I have pointed out, strong AI is emerging from many diverse efforts and will be deeply integrated 
into our civilization's infrastructure. Indeed, it will be intimately embedded in our bodies and brains. As such, it will 
reflect our values because it will be us. Attempts to control these technologies via secretive government programs, 
along with inevitable underground development, would only foster an unstable environment in which the dangerous 
applications would be likely to become dominant. 
DecentraIization.
One profound trend already well under way that will provide greater stability is the movement from 
centralized technologies to distributed ones and from the real world to the virtual world discussed above. Centralized 
technologies involve an aggregation of resources such as people (for example, cities, buildings), energy (such as 
nuclear-power plants, liquid-natural-gas and oil tankers, energy pipelines), transportation (airplanes, trains), and other 
items. Centralized technologies are subject to disruption and disaster. They also tend to be inefficient, wasteful, and 
harmful to the environment. 
Distributed technologies, on the other hand, tend to be flexible, efficient, and relatively benign in their 
environmental effects. The quintessential distributed technology is the Internet. The Internet has not been substantially 
disrupted to date, and as it continues to grow, its robustness and resilience continue to strengthen. If any hub or 
channel does go down, information simply routes around it.