ENGINES OF DESTRUCTION
- The Threat from the Machines
- Engines of Power
- Trustworthy Systems
- Tactics for the Assembler Breakthrough
- Is Success Possible?
- References for Chapter 11
Nor do I doubt if the most formidable armies ever heere upon earth
is a sort of soldiers who for their smallness are not visible.
- Sir WILLIAM PERRY, on microbes, 1640
REPLICATING assemblers and thinking
machines pose basic threats to people and to life on Earth. Today's organisms
have abilities far from the limits of the possible, and our machines are
evolving faster than we are. Within a few decades they seem likely to surpass
us. Unless we learn to live with them in safety, our future will likely
be both exciting and short. We cannot hope to foresee all the problems ahead,
yet by paying attention to the big, basic issues, we can perhaps foresee
the greatest challenges and get some idea of how to deal with them.
Entire books will no doubt be written on the coming social upheavals: What
will happen to the global order when assemblers and automated
engineering eliminate the need for most international trade? How will
society change when individuals can live indefinitely? What will we do when
replicating assemblers can make almost anything without human labor? What
will we do when AI systems can think faster than humans? (And before they
jump to the conclusion that people will despair of doing or creating anything,
the authors may consider how runners regard cars, or how painters regard
In fact, authors have already foreseen and discussed several of these issues.
Each is a matter of uncommon importance, but more fundamental than any of
them is the survival of life and liberty. After all, if life or liberty
is obliterated, then our ideas about social problems will no longer matter.
The Threat from the Machines
In Chapter 4, I described some of what
replicating assemblers will do for us if we handle them properly. Powered
by fuels or sunlight, they will be able to make almost anything (including
more of themselves) from common materials.
Living organisms are also powered by fuels or sunlight, and also make more
of themselves from ordinary materials. But unlike assembler-based systems,
they cannot make "almost anything".
Genetic evolution has limited
life to a system based on DNA,
RNA, and ribosomes, but memetic evolution
will bring life-like machines based on nanocomputers and assemblers. I have
already described how assembler-built molecular machines will differ from
the ribosome-built machinery of
life. Assemblers will be able to build all that ribosomes can, and more;
assembler-based replicators will therefore be able to do all that life can,
and more. From an evolutionary point of view, this poses an obvious threat
to otters, people, cacti, and ferns - to the rich fabric of the biosphere
and all that we prize.
The early transistorized computers soon beat the most advanced vacuum-tube
computers because they were based on superior devices. For the same reason,
early assembler-based replicators could beat the most advanced modern organisms.
"Plants" with "leaves" no more efficient than today's
solar cells could out-compete real
plants, crowding the biosphere with an inedible foliage. Tough, omnivorous
"bacteria" could out-compete
real bacteria: they could spread like blowing pollen, replicate swiftly,
and reduce the biosphere to dust in a matter of days. Dangerous replicators
could easily be too tough, small, and rapidly spreading to stop - at least
if we made no preparation. We have trouble
enough controlling viruses and fruit flies.
Among the cognoscenti of nanotechnology,
this threat has become known as the "gray goo problem." Though
masses of uncontrolled replicators need not be gray or gooey, the term "gray
goo" emphasizes that replicators able to obliterate life might be less
inspiring than a single species of crabgrass. They might be "superior"
in an evolutionary sense, but this need not make them valuable. We have
evolved to love a world rich in living things, ideas, and diversity, so
there is no reason to value gray goo merely because it could spread. Indeed,
if we prevent it we will thereby prove our evolutionary superiority.
The gray goo threat makes one thing perfectly clear: we cannot afford certain
kinds of accidents with replicating assemblers.
In Chapter 5, I described some of what
advanced AI systems will do for us, if we handle them properly. Ultimately,
they will embody the patterns of thought and make them flow at a pace no
mammal's brain can match. AI systems that work together as people do will
be able to out-think not just individuals, but whole societies. Again, the
evolution of genes has left life stuck. Again, the evolution of memes by
human beings - and eventually by machines - will advance our hardware far
beyond the limits of life. And again, from an evolutionary point of view
this poses an obvious threat.
Knowledge can bring power, and power can bring knowledge. Depending on their
natures and their goals, advanced AI systems might accumulate enough knowledge
and power to displace us, if we don't prepare properly. And as with replicators,
mere evolutionary "superiority" need not make the victors better
than the vanquished by any standard but brute competitive ability.
This threat makes one thing perfectly clear: we need to find ways to live
with thinking machines, to make them law-abiding citizens.
Engines of Power
Certain kinds of replicators and AI systems may confront us with forms of
hardware capable of swift, effective, independent action. But the novelty
of this threat - coming from the machines themselves - must not blind us
to a more traditional danger. Replicators and AI systems can also serve
as great engines of power, if wielded freely by sovereign states.
Throughout history, states have developed technologies to extend their military
power, and states will no doubt play a dominant role in developing replicators
and AI systems. States could use replicating assemblers to build arsenals
of advanced weapons, swiftly, easily, and in vast quantity. States could
use special replicators directly to wage a sort of germ warfare - one made
vastly more practical by programmable, computer-controlled "germs."
Depending on their skills, AI systems
could serve as weapon designers, strategists, or fighters. Military
funds already support research in both molecular
technology and artificial
States could use assemblers or advanced AI systems to achieve sudden, destabilizing
breakthroughs. I earlier discussed reasons for expecting that the advent
of replicating assemblers will bring relatively sudden changes: Able to
replicate swiftly, they could become abundant in a matter of days. Able
to make almost anything, they could be programmed to duplicate existing
weapons, but made from superior materials. Able to work with standard, well-understood
components (atoms) they could suddenly
build things designed in anticipation of the assembler breakthrough. These
results of design-ahead could include programmable germs and other nasty
novelties. For all these reasons, a state that makes the assembler breakthrough
could rapidly create a decisive military force - if not literally overnight,
then at least with unprecedented speed.
States could use advanced AI systems to similar ends. Automated engineering
systems will facilitate design-ahead and speed assembler development. Al
systems able to build better AI systems will allow an explosion of capability
with effects hard to anticipate. Both AI systems and replicating assemblers
will enable states to expand their military capabilities by orders of magnitude
in a brief time.
Replicators can be more potent than nuclear weapons: to devastate Earth
with bombs would require masses of exotic hardware and rare isotopes, but
to destroy all life with replicators would require only a single speck made
of ordinary elements. Replicators give nuclear war some company as a potential
cause of extinction, giving a broader context to extinction as a moral concern.
Despite their potential as engines of destruction, nanotechnology and AI
systems will lend themselves to more subtle uses than do nuclear weapons.
A bomb can only blast things, but nanomachines and AI systems could be used
to infiltrate, seize, change, and govern a territory or a world. Even the
most ruthless police have no use for nuclear weapons, but they do have use
for bugs, drugs, assassins, and other flexible engines of power. With advanced
technology, states will be able to consolidate their power over people.
Like genes, memes, organisms, and hardware, states have evolved. Their institutions
have spread (with variations) through growth, fission, imitation, and conquest.
States at war fight like beasts, but using citizens as their bones, brains,
and muscle. The coming breakthroughs will confront states with new pressures
and opportunities, encouraging sharp changes in how states behave. This
naturally gives cause for concern. States have, historically, excelled at
slaughter and oppression.
In a sense, a state is simply the sum of the people making up its organizational
apparatus: their actions add up to make its actions. But the same might
be said of a dog and its cells, though a dog is clearly more than just a
clump of cells. Both dogs and states are evolved systems, with structures
that affect how their parts behave. For thousands of years, dogs have evolved
largely to please people, because they have survived and reproduced at human
whim. For thousands of years, states have evolved under other selective
pressures. Individuals have far more power over their dogs than they do
over "their" states. Though states, too, can benefit from pleasing
people, their very existence has depended on their capability for using
people, whether as leaders, police, or soldiers.
It may seem paradoxical to say that people have limited power over states:
After all, aren't people behind a state's every action? But in democracies,
heads of state bemoan their lack of power, representatives bow to interest
groups, bureaucrats are bound by rules, and voters, allegedly in charge,
curse the whole mess. The state acts and people affect it, yet no one can
claim to control it. In totalitarian states, the apparatus of power has
a tradition, structure, and inner logic that leaves no one free, neither
the rulers nor the ruled. Even kings had to act in ways limited by the traditions
of monarchy and the practicalities of power, if they were to remain kings.
States are not human, though they are made of humans.
Despite this, history shows that change is possible, even change for the
better. But changes always move from one semi-autonomous, inhuman system
to another - equally inhuman but perhaps more humane. In our hope for improvements,
we must not confuse states that wear a human face with states that have
Describing states as quasi-organisms captures only one aspect of a complex
reality, yet it suggests how they may evolve in response to the coming breakthroughs.
The growth of government power, most spectacular in totalitarian countries,
suggests one direction.
States could become more like organisms by dominating their parts more completely.
Using replicating assemblers, states could fill the human environment with
miniature surveillance devices. Using an abundance of speech-understanding
AI systems, they could listen to everyone without employing half the population
as listeners. Using nanotechnology like that proposed for cell
repair machines, they could cheaply tranquilize, lobotomize, or otherwise
modify entire populations. This would simply extend an all too familiar
pattern. The world already holds governments that spy, torture, and drug;
advanced technology will merely extend the possibilities.
But with advanced technology, states need not control people - they could
instead simply discard people. Most people in most states, after
all, function either as workers, larval workers, or worker-rearers, and
most of these workers make, move, or grow things. A state with replicating
assemblers would not need such work. What is more, advanced AI systems could
replace engineers, scientists, administrators, and even leaders. The combination
of nanotechnology and advanced AI will make possible intelligent, effective
robots; with such robots, a state could prosper while discarding anyone,
or even (in principle) everyone.
The implications of this possibility depend on whether the state exists
to serve the people, or the people exist to serve the state.
In the first case, we have a state shaped by human beings to serve general
human purposes; democracies tend to be at least rough approximations to
this ideal. If a democratically controlled government loses its need for
people, this will basically mean that it no longer needs to use people as
bureaucrats or taxpayers. This will open new possibilities, some of which
may prove desirable.
In the second case, we have a state evolved to exploit human beings, perhaps
along totalitarian lines. States have needed people as workers because human
labor has been the necessary foundation of power. What is more, genocide
has been expensive and troublesome to organize and execute. Yet, in this
century totalitarian states have slaughtered their citizens by the millions.
Advanced technology will make workers unnecessary and genocide easy. History
suggests that totalitarian states may then eliminate people wholesale. There
is some consolation in this. It seems likely that a state willing and able
to enslave us biologically would instead simply kill us.
The threat of advanced technology in the hands of governments makes one
thing perfectly clear: we cannot afford to have an oppressive state take
the lead in the coming breakthroughs.
The basic problems I have outlined are obvious: in the future, as in the
past, new technologies will lend themselves to accidents and abuse. Since
replicators and thinking machines will bring great new powers, the potential
for accidents and abuse will likewise be great. These possibilities pose
genuine threats to our lives.
Most people would like a chance to live and be free to choose how to live.
This goal may not sound too utopian, at least in some parts of the world.
It doesn't mean forcing everyone's life to fit some grand scheme; it chiefly
means avoiding enslavement and death. Yet, like the achievement of a utopian
dream, it will bring a future of wonders.
Given these life-and-death problems and this general goal, we can consider
what measures might help us succeed. Our strategy must involve people, principles,
and institutions, but it must also rest on tactics which inevitably will
To use such powerful technologies in safety, we must make hardware we can
trust. To have trust, we must be able to judge technical facts accurately,
an ability that will in turn depend partly on the quality of our institutions
for judgment. More fundamentally, though, it will depend on whether trustworthy
hardware is physically possible. This is a matter of the reliability of
components and of systems.
We can often make reliable components, even without assemblers to help.
" Reliable" doesn't mean "indestructible" - anything
will fail if placed close enough to a nuclear blast. It doesn't even mean
"tough" - a television set may be reliable, yet not survive being
bounced off a concrete floor. Rather, we call something reliable when we
can count on it to work as designed.
A reliable component need not be a perfect embodiment of a perfect design:
it need only be a good enough embodiment of a cautious enough design. A
bridge engineer may be uncertain about the strength of winds, the weight
of traffic, and the strength of steel, but by assuming high winds, heavy
traffic, and weak steel, the engineer can design a bridge that will stand.
Unexpected failures in components commonly stem from material flaws. But
assemblers will build components that have a negligible number of their
atoms out of place - none, if need be.
This will make them perfectly uniform and in a limited sense perfectly reliable.
Radiation will still cause damage, though, because a
cosmic ray can unexpectedly knock atoms loose from anything. In a small
enough component (even in a modern computer memory device), a single particle
of radiation can cause a failure.
[Addition to Web version of Engines of Creation:
A reader of this web version has noted a problem with the math in the following
example. As an example of the value of hypertext as discussed in Chapter
14 and in Eric Drexler's essay "Hypertext
Publishing and the Evolution of Knowledge", this correspondence
about the calculation can be read here.
But systems can work even when their parts fail; the key is redundancy.
Imagine a bridge suspended from cables that fail randomly, each breaking
about once a year at an unpredictable time. If the bridge falls when a cable
breaks, it will be too dangerous to use. Imagine, though, that a broken
cable takes a day to fix (because skilled repair crews with spare cables
are on call), and that, though it takes five cables to support the bridge,
there are actually six. Now if one cable breaks, the bridge will
still stand. By clearing traffic and then replacing the failed cable, the
bridge operators can restore safety. To destroy this bridge, a second cable
must break in the same day as the first. Supported by six cables, each having
a one-in-365 daily chance of breaking, the bridge will likely last about
While an improvement, this remains terrible. Yet a bridge with ten cables
(five needed, five extra) will fall only if six cables break on the same
day: the suspension system is likely to last over ten million years. With
fifteen cables, the expected lifetime is over ten thousand times the age
of the Earth. Redundancy can bring an exponential explosion of safety.
Redundancy works best when the redundant components are truly independent.
If we don't trust the design process, then we must use components designed
independently; if a bomb, bullet, or cosmic ray may damage several neighboring
parts, then we must spread redundant parts more widely. Engineers who want
to supply reliable transportation between two islands shouldn't just add
more cables to a bridge. They should build two well-separated bridges using
different designs, then add a tunnel, a ferry, and a pair of inland airports.
Computer engineers also use redundancy. Stratus Computer Inc., for example,
makes a machine that uses four central
processing units (in two pairs) to do the work of one, but to do it
vastly more reliably. Each pair is continually checked for internal consistency,
and a failed pair can be replaced while its twin carries on.
An even more powerful form of redundancy
is design diversity.
In computer hardware, this means using several computers with different
designs, all working in parallel. Now redundancy can correct not just for
failures in a piece of hardware, but for errors in its design.
Much has been made of the problem of writing large, error-free programs;
many people consider such programs impossible to develop and debug. But
researchers at the UCLA Computer Science Department have shown that design
diversity can also be used in software: several programmers can tackle the
same problem independently, then all their programs can be run in parallel
and made to vote on the answer. This multiplies the cost of writing and
running the program, but it makes the resulting software system resistant
to the bugs that appear in some of its parts.
We can use redundancy to control replicators.
Just as repair machines that compare multiple DNA strands will be able to
correct mutations in a cell's genes, so replicators that compare multiple
copies of their instructions (or that use other
effective error-correcting systems) will be able to resist mutation
in these "genes." Redundancy can again bring an exponential explosion
We can build systems that are extremely reliable, but this will entail costs.
Redundancy makes systems heavier, bulkier, more expensive, and less efficient.
Nanotechnology, though, will make most things far lighter, smaller, cheaper,
and more efficient to begin with. This will make redundancy and reliability
Today, we are seldom willing to pay for the safest possible systems; we
tolerate failures more-or-less willingly and seldom consider the real limits
of reliability. This biases judgments of what can be achieved. A psychological
factor also distorts our sense of how reliable things can be made: failures
stick in our minds, but everyday successes draw little attention. The media
amplify this tendency by reporting the most dramatic failures from around
the world, while ignoring the endless and boring successes. Worse yet, the
components of redundant systems may fail in visible ways, stirring alarums:
imagine how the media would report a snapped bridge cable, even if the bridge
were the super-safe fifteen-cable model described above. And since each
added redundant component adds to the chance of a component failure, a system's
reliability can seem worse even as it approaches perfection.
Appearances aside, redundant systems made of abundant, flawless components
can often be made almost perfectly reliable. Redundant systems spread over
wide enough spaces will survive even bullets and bombs.
But what about design errors? Having a dozen redundant parts will do no
good if they share a fatal error in design. Design diversity is one answer;
good testing is another. We can reliably evolve good designs without being
reliably good designers: we need only be good at testing, good at tinkering,
and good at being patient. Nature has evolved working molecular machinery
through entirely mindless tinkering and testing. Having minds, we can do
as well or better.
We will find it easy to design reliable hardware if we can develop reliable
automated engineering systems. But this raises the wider issue of developing
trustworthy artificial intelligence systems. We will have little trouble
making AI systems with reliable hardware, but what about their software?
Like present AI systems and human minds, advanced AI systems will be synergistic
combinations of many simpler parts. Each part will be more specialized and
less intelligent than the system as a whole. Some parts will look for patterns
in pictures, sounds, and other data and suggest what they might mean. Other
parts will compare and judge the suggestions of these parts. Just as the
pattern recognizers in the human visual system suffer from errors and optical
illusions, so will the pattern recognizers in AI systems. (Indeed, some
advanced machine vision systems already suffer from familiar optical illusions.)
And just as other parts of the human mind can often identify and compensate
for illusions, so will other parts of AI systems.
As in human minds, intelligence will
involve mental parts that make shaky guesses and other parts that discard
most of the bad guesses before they draw much attention or affect important
decisions. Mental parts that reject action ideas on ethical grounds correspond
to what we call a conscience. AI systems with many parts will have room
for redundancy and design diversity, making reliability possible.
A genuine, flexible AI system must evolve ideas. To do this, it must find
or form hypotheses, generate variations, test them, and then modify or discard
those found inadequate. Eliminating any of these abilities would make it
stupid, stubborn, or insane ("Durn machine can't think and won't learn
from its mistakes - junk it!"). To avoid becoming trapped by initial
misconceptions, it will have to consider conflicting views, seeing how well
each explains the data, and seeing whether one view can explain another.
Scientific communities go through a similar process. And in a paper called
"The Scientific Community Metaphor,"
William A. Kornfeld and Carl Hewitt of the MIT Artificial Intelligence Laboratory
suggest that AI researchers model their programs still more closely on the
evolved structure of the scientific community. They point to the pluralism
of science, to its diversity of competing proposers, supporters, and critics.
Without proposers, ideas cannot appear; without supporters, they cannot
grow; and without critics to weed them, bad ideas can crowd out the good.
This holds true in science, in technology, in AI systems, and among the
parts of our own minds.
Having a world full of diverse and redundant proposers, supporters, and
critics is what makes the advance of science and technology reliable. Having
more proposers makes good proposals more common; having more critics makes
bad proposals more vulnerable. Better, more numerous ideas are the result.
A similar form of redundancy can help AI systems to develop sound ideas.
People sometimes guide their actions by standards of truth and ethics, and
we should be able to evolve AI systems that do likewise, but more reliably.
Able to think a million times faster than us, they will have more time for
second thoughts. It seems that AI systems
can be made trustworthy, at least by human standards.
I have often compared AI systems to individual human minds, but the resemblance
need not be close. A system that can mimic a person may need to be personlike,
but an automated engineering system probably doesn't. One
proposal (called an Agora system, after the Greek term for a meeting
and market place) would consist of many independent pieces of software that
interact by offering one another services in exchange for money. Most pieces
would be simpleminded specialists, some able to suggest a design change,
and others able to analyze one. Much as Earth's ecology has evolved extraordinary
organisms, so this computer economy could evolve extraordinary designs -
and perhaps in a comparably mindless fashion. What is more, since the system
would be spread over many machines and have parts written by many people,
it could be diverse, robust, and hard for any group to seize and abuse.
Eventually, one way or another, automated engineering systems will be able
to design things more reliably than any
group of human engineers can today. Our challenge will be to design
them correctly. We will need human institutions that reliably develop reliable
Human institutions are evolved artificial systems, and they can often solve
problems that their individual members cannot. This makes them a sort of
"artificial intelligence system." Corporations, armies, and research
laboratories all are examples, as are the looser structures of the market
and the scientific community. Even governments may be seen as artificial
intelligence systems - gross, sluggish, and befuddled, yet superhuman in
their sheer capability. And what are constitutional checks and balances
but an attempt to increase a government's reliability through institutional
diversity and redundancy? When we build intelligent machines, we will use
them to check and balance one another.
By applying the sane principles, we may be able to develop reliable, technically
oriented institutions having strong checks and balances, then use these
to guide the development of the systems we will need to handle the coming
Tactics for the Assembler Breakthrough
Some force in the world (whether trustworthy or not) will take the lead
in developing assemblers; call it the "leading force." Because
of the strategic importance of assemblers, the leading force will presumably
be some organization or institution that is effectively controlled by some
government or group of governments. To simplify matters, pretend for the
moment that we (the good guys, attempting to be wise) can make policy for
the leading force. For citizens of democratic states, this seems a good
attitude to take.
What should we do to improve our chances of reaching a future worth living
in? What can we do?
We can begin with what must not happen: we must not let a single
replicating assembler of the wrong kind be loosed on an unprepared world.
Effective preparations seem possible (as I will describe), but it seems
that they must be based on assembler-built systems that can be built only
after dangerous replicators have already become possible. Design-ahead can
help the leading force prepare, yet even vigorous, foresighted action seems
inadequate to prevent a time of danger. The reason is straightforward: dangerous
replicators will be far simpler to design than systems that can thwart them,
just as a bacterium is far simpler than an immune system. We will need tactics
for containing nanotechnology while we learn how to tame it.
One obvious tactic is isolation: the leading force will be able to contain
replicator systems behind multiple
walls or in laboratories in space. Simple replicators will have no intelligence,
and they won't be designed to escape and run wild. Containing them seems
no great challenge.
Better yet, we will be able to design replicators that can't escape
and run wild. We can build them with counters (like those in cells) that
limit them to a fixed number of replications. We can build them to have
requirements for special synthetic "vitamins," or for bizarre
environments found only in the laboratory. Though replicators could be made
tougher and more voracious than any modern pests, we can also make them
useful but harmless. Because we will design them from scratch, replicators
need not have the elementary survival skills that evolution has built into
Further, they need not be able to evolve. We can give replicators redundant
copies of their "genetic" instructions, along with repair mechanisms
to correct any mutations. We can design them to stop working long before
enough damage accumulates to make a lasting mutation a significant possibility.
Finally, we can design them in ways that would hamper evolution even if
mutations could occur.
Experiments show that most computer programs (other
than specially designed AI programs, such as Dr. Lenat's EURISKO)
seldom respond to mutations by changing slightly; instead, they simply fail.
Because they cannot vary in useful ways, they cannot evolve. Unless they
are specially designed, replicators directed by nanocomputers will share
this handicap. Modern organisms are fairly good at evolving partly because
they descend from ancestors that evolved. They are evolved to evolve; this
is one reason for the complexities of sexual reproduction and the shuffling
of chromosome segments during the production of sperm and egg cells. We
can simply neglect to give replicators similar talents.
It will be easy for the leading force to make replicating assemblers useful,
harmless, and stable. Keeping assemblers from being stolen and abused is
a different and greater problem, because it will be a game played against
intelligent opponents. As one tactic, we can reduce the incentive to steal
assemblers by making them available in safe forms. This will also reduce
the incentive for other groups to develop assemblers independently. The
leading force, after all, will be followed by trailing forces.
In Chapter 4, I described how a system
of assemblers in a vat could build an excellent rocket engine. I also pointed
out that we will be able to make assembler systems that act like seeds,
absorbing sunlight and ordinary materials and growing to become almost anything.
These special-purpose systems will not replicate themselves, or will do
so only a fixed number of times. They will make only what they were programmed
to make, when they are told to make it. Anyone lacking special
assembler-built tools would be unable to reprogram them to serve other purposes.
Using limited assemblers of this sort, people will be able to make as much
as they want of whatever they want, subject to limits built into the machines.
If none is programmed to make nuclear weapons, none will; if none is programmed
to make dangerous replicators, none will. If some are programmed to make
houses, cars, computers, toothbrushes, and whatnot, then these products
can become cheap and abundant. Machines built by limited assemblers will
enable us to open space, heal the biosphere, and repair human cells. Limited
assemblers can bring almost unlimited wealth to the people of the world.
This tactic will ease the moral pressure to make unlimited assemblers available
immediately. But limited assemblers will still leave legitimate needs unfulfilled.
Scientists will need freely programmable assemblers to conduct studies;
engineers will need them to test designs. These needs can be served by sealed
Sealed Assembler Laboratories
Picture a computer accessory the size of your thumb, with a state-of-the-art
plug on its bottom. Its surface looks like boring gray plastic, imprinted
with a serial number, yet this sealed assembler lab is an assembler-built
object that contains many things. Inside, sitting above the plug, is a large
nanoelectronic computer running advanced molecular-simulation software (based
on the software developed during assembler development). With the assembler
lab plugged in and turned on, your assembler-built home computer displays
a three-dimensional picture of whatever the lab computer is simulating,
representing atoms as colored spheres. With a joystick, you can direct the
simulated assembler arm to build things. Programs can move the arm faster,
building elaborate structures on the screen in the blink of an eye. The
simulation always works perfectly, because the nanocomputer
cheats: as you make the simulated arm move simulated molecules, the computer
directs an actual arm to move actual molecules. It
then checks the results whenever needed to correct its calculations.
The end of this thumb-sized object holds a sphere built in many concentric
layers. Fine wires carry power and signals through the layers; these let
the nanocomputer in the base communicate with the devices at the sphere's
center. The outermost layer consists of sensors. Any attempt to remove or
puncture it triggers a signal to a layer near the core. The next layer in
is a thick spherical shell of prestressed diamond composite, with its outer
layers stretched and its inner layers compressed. This surrounds a layer
of thermal insulator which in turn surrounds a peppercorn-sized spherical
shell made up of microscopic, carefully arranged blocks of metal and oxidizer.
These are laced with electrical igniters. The outer sensor layer, if punctured,
triggers these igniters. The metal-and-oxidizer demolition charge then burns
in a fraction of a second, producing a gas of metal oxides denser than water
and almost as hot as the surface of the Sun. But the blaze is tiny; it swiftly
cools, and the diamond sphere confines its great pressure.
This demolition charge surrounds a smaller composite shell, which surrounds
another layer of sensors, which can also trigger the demolition charge.
These sensors surround the cavity which contains the actual sealed assembler
These elaborate precautions justify the term "sealed." Someone
outside cannot open the lab space without destroying the contents, and no
assemblers or assembler-built structures can escape from within. The
system is designed to let out information, but not dangerous replicators
or dangerous tools. Each sensor layer
consists of many redundant layers of sensors, each intended
to detect any possible penetration, and each making up for possible flaws
in the others. Penetration, by triggering the demolition charge, raises
the lab to a temperature beyond the melting point of all possible substances
and makes the survival of a dangerous device impossible. These protective
mechanisms all gang up on something about a millionth their size - that
is, on whatever will fit in the lab, which provides a spherical work space
no wider than a human hair.
Though small by ordinary standards, this work space holds room enough for
millions of assemblers and thousands of trillions of atoms. These sealed
labs will let people build and test devices, even voracious replicators,
in complete safety. Children will use the atoms inside them as a construction
set with almost unlimited parts. Hobbyists will exchange programs for building
various gadgets. Engineers will build and test new nanotechnologies. Chemists,
materials scientists, and biologists will build apparatus and run experiments.
In labs built around biological samples, biomedical engineers will develop
and test early cell repair machines.
In the course of this work, people will naturally develop useful designs,
whether for computer circuits, strong materials, medical devices, or whatever.
After a public review of their safety, these things could be made available
outside the sealed labs by programming limited assemblers to make them.
Sealed labs and limited assemblers will form a complementary pair: The first
will let us invent freely; the second will let us enjoy the fruits of our
invention safely. The chance to pause between design and release will help
us avoid deadly surprises.
Sealed assembler labs will enable the whole of society to apply its creativity
to the problems of nanotechnology. And this will speed our preparations
for the time when an independent force learns how to build something nasty.
In another tactic for buying time, the leading force can attempt to burn
the bridge it built from bulk to molecular technology. This means destroying
the records of how the first assemblers were made (or making the records
thoroughly inaccessible). The leading force may be able to develop the first,
crude assemblers in such a way that no one knows the details of more than
a small fraction of the whole system. Imagine that we develop assemblers
by the route outlined in Chapter 1. The
protein machines that we use to build the first crude assemblers will then
promptly become obsolete. If we destroy
the records of the protein designs, this will hamper efforts to duplicate
them, yet will not hamper further progress in nanotechnology.
If sealed labs and limited assemblers are widely available, people will
have little scientific or economic motivation to redevelop nanotechnology
independently, and burning the bridge from bulk
technology will make independent development more difficult. Yet these
can be no more than delaying tactics. They won't stop independent development;
the human urge for power will spur efforts which will eventually succeed.
Only detailed, universal policing on a totalitarian scale could stop independent
development indefinitely. If the policing were conducted by anything like
a modern government, this would be a cure roughly as dangerous as the disease.
And even then, would people maintain perfect vigilance forever?
It seems that we must eventually learn to live in a world with untrustworthy
replicators. One sort of tactic would be to hide behind a wall or to run
far away. But these are brittle methods: dangerous replicators might breach
the wall or cross the distance, and bring utter disaster. And, though walls
can be made proof against small replicators, no
fixed wall will be proof against large-scale, organized malice. We will
need a more robust, flexible approach.
It seems that we can build nanomachines that act somewhat like the white
blood cells of the human immune system: devices that can fight not just
bacteria and viruses, but dangerous replicators of all sorts. Call an automated
defense of this sort an active
shield, to distinguish it from a fixed wall.
Unlike ordinary engineering systems, reliable active shields must do more
than just cope with nature or clumsy users. They must also cope with a far
greater challenge - with the entire range of threats that intelligent forces
can design and build under prevailing circumstances. Building and improving
prototype shields will be akin to running both sides of an arms race on
a laboratory scale. But the goal here will be to seek the minimum requirements
for a defense that reliably prevails.
In Chapter 5, I described how Dr. Lenat
and his EURISKO program evolved successful fleets to fight according to
the rules of a naval-warfare simulation game. In a similar way, we can make
into a game the deadly serious effort to develop reliable shields, using
sealed assembler labs of various sizes as playing fields. We can turn loose
a horde of engineers, computer hackers, biologists, hobbyists, and automated
engineering systems, all invited to pit their systems against one another
in games limited only by the initial conditions, the laws of nature, and
the walls of the sealed labs. These competitors will evolve threats and
shields in an open-ended series of microbattles. When replicating assemblers
have brought abundance, people will have time enough for so important a
game. Eventually we can test promising shield systems in Earthlike environments
in space. Success will make possible a system able to protect human life
and Earth's biosphere from the worst that a fistful of loose replicators
Is Success Possible?
With our present uncertainties, we cannot yet describe either threats or
shields with any accuracy. Does this mean we can't have any confidence that
effective shields are possible? Apparently we can; there is a difference,
after all, between knowing that something is possible and knowing how to
do it. And in this case, the world holds examples of analogous successes.
There is nothing fundamentally novel about defending against invading replicators;
life has been doing it for ages. Replicating assemblers, though unusually
potent, will be physical systems not unlike those we already know. Experience
suggests that they can be controlled.
Viruses are molecular machines that invade cells; cells use molecular machines
(such as restriction enzymes and antibodies) to defend against them. Bacteria
are cells that invade organisms; organisms use cells (such as white blood
cells) to defend against them. Similarly, societies use police to defend
against criminals and armies to defend against invaders. On a less physical
level, minds use meme systems such
as the scientific method to defend against nonsense, and societies use institutions
such as courts to defend against the power of other institutions.
The biological examples in the last paragraph show that even after a billion-year
arms race, molecular machines have maintained successful defenses against
molecular replicators. Failures have been common too, but the successes
do indicate that defense is possible. These successes suggest that we can
indeed use nanomachines to defend against nanomachines. Though assemblers
will bring many advances, there seems no reason why they should permanently
tip the balance against defense.
The examples given above - some involving viruses, some involving institutions
- are diverse enough to suggest that successful defense rests on general
principles. One might ask, Why do all these defenses succeed? But turn the
question around: Why should they fail? Each conflict pits similar systems
against each other, giving the attacker
no obvious advantage. In each conflict, moreover, the attacker faces
a defense that is well established. The defenders fight on home
ground, giving them advantages such as prepared positions, detailed local
knowledge, stockpiled resources, and abundant allies - when the immune system
recognizes a germ, it can mobilize the resources of an entire body. All
these advantages are general and basic, having little to do with the details
of a technology. We can give our active shields the same advantages over
dangerous replicators. And they need not sit idle while dangerous weapons
are amassed, any more than the immune system sits idle while bacteria multiply.
It would be hard to predict the outcome of an open-ended arms race between
powers equipped with replicating assemblers. But before this situation can
arise, the leading force seems likely to acquire a temporary but overwhelming
military advantage. If the outcome of an arms race is in doubt, then the
leading force will likely use its strength to ensure that no opponents are
allowed to catch up. If it does so, then active shields will not have to
withstand attacks backed by the resources of half a continent or half a
solar system; they will instead be like a police force or an immune system,
facing attacks backed only by whatever resources can be gathered in secret
within the protected territory.
In each case of successful defense that I cited above, the attackers and
the shields have developed through broadly similar processes. The immune
system, shaped by genetic evolution, meets threats also shaped by genetic
evolution. Armies, shaped by human minds, also meet similar threats. Likewise,
both active shields and dangerous replicators will be shaped by memetic
evolution. But if the leading force can develop automated engineering systems
that work a millionfold faster than human engineers, and if it can use them
for a single year, then it can build active shields based on a million years'
worth of engineering advance. With such systems we may be able to explore
the limits of the possible well enough to build a reliable shield against
all physically possible threats.
Even without our knowing the details of the threats and the shields, there
seems reason to believe that shields are possible. And the examples of memes
controlling memes and of institutions controlling institutions also suggest
that AI systems can control AI systems.
In building active shields, we will be able to use the power of replicators
and AI systems to multiply the traditional advantages of the defending force:
we can give it overwhelming strength through abundant, replicator-built
hardware with designs based on the equivalent of a million-year lead in
technology. We can build active shields having strength and reliability
that will put past systems to shame.
Nanotechnology and artificial intelligence could bring the ultimate tools
of destruction, but they are not inherently destructive. With care, we can
use them to build the ultimate tools of peace.
[ Table of Contents
[ Previous Section |
Next Section ]
© Copyright 1986, K. Eric Drexler. All rights reserved.
Published and maintained by Russell Whitaker.
Last updated: 23 September 1996