- VANNEVAR BUSH
SOCIETY NEEDS BETTER WAYS to understand technology - this has long been obvious. The challenges ahead simply make our need more urgent.
The promise of technology lures us onward, and the pressure of competition makes stopping virtually impossible. As the technology race quickens, new developments sweep toward us faster, and a fatal mistake grows more likely. We need to strike a better balance between our foresight and our rate of advance. We cannot do much to slow the growth of technology, but we can speed the growth of foresight. And with better foresight, we will have a better chance to steer the technology race in safe directions.
Various approaches to guiding technology have been suggested. "The people must control technology" is a plausible slogan, but it has two possible meanings. If it means that we must make technology serve human needs, then it makes good sense. But if it means that the people as a whole must make technical decisions, then it makes very little sense. The electorate cannot judge the intricate links between technology, economy, environment, and life; people lack the needed knowledge. The people themselves agree: according to a U.S. National Science Foundation survey, 85 percent of U.S. adults believe that most citizens lack the knowledge needed to choose which technologies to develop. The public generally leaves technical judgments to technical experts.
Unfortunately, leaving judgment to experts causes problems. In Advice and Dissent, Primack and von Hippel point out that "to the extent that the Administration can succeed in keeping unfavorable information quiet and the public confused, the public welfare can be sacrificed with impunity to bureaucratic convenience and private gain." Regulators suffer more criticism when a new drug causes a single death than they do when the absence of a new drug causes a thousand deaths. They misregulate accordingly. Military bureaucrats have a vested interest in spending money, hiding mistakes, and continuing their projects. They mismanage accordingly. This sort of problem is so basic and natural that more examples are hardly needed. Everywhere, secrecy and fog make bureaucrats more comfortable; everywhere, personal convenience warps factual statements on matters of public concern. As technologies grow more complex and important, this pattern grows more dangerous.
Some authors consider rule by secretive technocrats to be virtually inevitable. In Creating Alternative Futures, Hazel Henderson argues that complex technologies "become inherently totalitarian" (her italics) because neither voters nor legislators can understand them. In The Human Future Revisited, Harrison Brown likewise argues that the temptation to bypass democratic processes in solving complex crises brings the danger "that if industrial civilization survives it will become increasingly totalitarian in nature." If this were so, it would likely mean our doom: we cannot stop the technology race, and a world of totalitarian states based on advanced technology needing neither workers nor soldiers might well discard most of the population.
Fortunately, democracy and liberty have met comparable challenges before. States grew too complex for direct democracy, but representative government evolved. State power threatened to crush liberty, but the rule of law evolved. Technology has grown complex, but this gives us no reason to ignore the people, discard the law, and hail a dictator. We need ways to handle technical complexity in a democratic framework, using experts as instruments to clarify our vision without giving them control of our lives. But technical experts today are mired in a system of partisan feuding.
To gain influence in our mass democracy, groups try to outshout one another. When their views have corporate appeal, they take them to the public through advertising campaigns. When their views have pork-barrel appeal, they take them to legislatures through lobbying. When their views have dramatic appeal, they take them to the public through media campaigns. Groups promote their pet experts, the battle goes public, and quiet scientists and engineers are drowned in the clamor.
As the public conflict grows, people come to doubt expert pronouncements. They judge statements the obvious way, by their source. ("Of course she claims oil spills are harmless - she works for Exxon." "Of course he says Exxon lies - he works for Nader.")
When established experts lose credibility, demagogues can join the battle on an equal footing. Reporters - eager for controversy, striving for fairness, and seldom guided by technical backgrounds-carry all sides straight to the public. Cautious statements by scrupulous scientists make little impression; other scientists see no choice but to adopt the demagogues' style. Debates become sharp and angry, divisions grow, and the smoke of battle obscures the facts. Paralysis or folly often follows.
Our greatest problem is how we handle problems. Debates rage over the safety of nuclear power, coal power, and chemical wastes. Well-meaning groups backed by impressive experts clash again and again over dull, technical facts - dull, that is, save for their importance: What are the effects of low-level radiation, and how likely is a reactor meltdown? What are the causes and effects of acid rain? How well could space-based defenses block missile attacks? Do five cases of leukemia within three miles of a particular waste dump show a deadly hazard, or merely the workings of chance?
Greater issues lie ahead: How safe is this replicator? Will this active shield system be safe and secure? Will this biostasis procedure be reversible? Can we trust this AI system?
Disputes about technical facts feed broader disputes about policy. People may have differing values (which would you rather have, encephalitis or pesticide poisoning?) but their views of relevant facts often differ still more. (How often do these mosquitoes carry encephalitis? How toxic is this pesticide?) When different views of boring facts lead to disagreements about important policies, people may wonder, "How can they oppose us on this vital issue unless they have bad motives?" Disputes over facts can thus turn potential allies against one another. This hampers our efforts to understand and solve our problems.
People have disputed facts for millennia; only the prominence of technical disputes is new. Societies have evolved methods for judging facts about people. These methods suggest how we might judge facts about technology.
They used trial by battle: opponents fought, and the law vindicated the victor.
They used compurgation: neighbors swore to the honesty of the accused; if enough swore, the charges were dropped.
They used trial by ordeal: in one, the accused was bound and thrown in a river; those who sank were innocent, those who floated, guilty.
They used judgment by secretive committees: the king's councilors would meet to judge and pass sentence as seemed fit. In England, they met in a room called the Star Chamber.
These methods supposedly determined who did what - the facts about human events - but all had serious shortcomings. Today we use similar methods to determine what causes what - the facts about science and technology:
We use trial by combat in the press: opponents fling sharp words until one side's case suffers political death. Unfortunately, this often resembles an endless feud.
We use compurgation: experts swear to certain facts; if enough swear the same, the facts are declared true.
We use judgment by secretive committees: selected experts meet to judge facts and recommend such actions as seem fit. In the United States, they often meet in committees of the National Academy of Sciences.
Trial by ordeal has passed from fashion, but combat in the press may well seem like torture to the quiet scientist with self-respect.
The English abolished Star Chamber proceedings in 1641, and they counted this a great achievement. Trial by combat, compurgation, and ordeal have likewise become history. We now value due process, at least when judging people.
Court procedures illustrate the principles of due process: Allegations must be specific. Both sides must have a chance to speak and confront each other, to rebut and cross-examine. The process must be public, to prevent hidden rot. Debate must proceed before a jury that both sides accept as impartial. Finally, a judge must referee the process and enforce the rules.
To see the value of due process, imagine its opposite: a process trampling all these principles would give one side a say and the other no chance to cross-examine or respond. It would meet in secret, allow vague smears, and lack a judge to enforce whatever rules might remain. Jurors would arrive with their decisions made. In short, it would resemble a lynch mob meeting in a locked barn - or a rigged committee drafting a report.
Experience shows the value of due process in judging facts about people; might it also be of value in judging facts about science and technology? Due process is a basic idea, not restricted to courts of law. Some AI researchers, for example, are building due-process principles into their computer programs. It seems that due process should be of use in judging technical facts.
In fact, the scientific literature - the chief forum of science - already embodies a form of due process: In good journals, scientific statements must be specific. In theory, given enough time and persistence, all sides may state their views in a dispute, since journals stand open to controversy. Though opponents may not meet face to face, they confront each other at a distance; they question and respond in slow motion, through letters and articles. Referees, like juries, evaluate evidence and reasoning. Editors, like judges, enforce rules of procedure. Publication keeps the debate open to public scrutiny.
In both journals and courts, conflicting ideas are pitted against one another under rules evolved to ensure a fair, orderly battle. These rules sometimes fail because they are broken or inadequate, but they are the best we have developed. Imperfect due process has proved better than no due process at all.
Why do scientists value refereed journals? Not because they trust all refereed journals, or trust everything printed in any one of them. Even the best due-process system won't grind out a stream of pure truth. Rather, they value refereed journals because they tend to reflect sound critical discussion. Indeed, they must: because journals compete with one another for papers, prestige, and readership, the best journals must be good indeed. Journals grind slowly, yet after enough rounds of publication and criticism they often grind out consensus.
Experience proves the value of both courts and journals. Their underlying similarity suggests that their value stems from a common source - due process. Due process can fail, but it is still the best approach known for finding the facts.
Today, courts and journals are not enough. Vital technical disputes go on and on because we have no rapid, orderly way to bring out the facts (and to delineate our ignorance). Courts are not suited to deal with technical questions. Journals are better, but they still have shortcomings. They took shape in a time of lower technology and slower advance, evolving to fit the limits of printing, the speed of mail, and the needs of academic science. But today, in a time when we desperately need better and swifter technical judgment, we find ourselves in a world that has telephones, jets, copiers, and express and electronic mail. Can we use modern technologies to speed technical debate?
Of course: scientists already use several approaches. Jets bring scientists from around the globe to conferences where papers are presented and discussed. But conferences handle controversy poorly; public decorum and tight schedules limit the vigor and depth of debate.
Scientists also join informal research networks linked by telephone, mail, computers, and copying machines; these accelerate exchange and discussion. But they are essentially private institutions. They, too, fail to provide a credible, public procedure for thrashing out differences.
Conferences, journals, and informal networks share some similar limitations. They typically focus on technical questions of scientific importance, rather than on technical questions of public-policy importance. Moreover, they typically focus on scientific questions. Journals tend to slight technological questions that lack intrinsic scientific interest; they often treat them as news items not worthy of checking by referees. Further, our present institutions lack any balanced way to present knowledge when it is still tangled in controversy. Though scientific review articles often present and weigh several sides, they do so from a single author's point of view.
All these shortcomings share a common source: scientific institutions evolved to advance science, not to sift facts for policy makers. These institutions serve their purpose well enough, but they serve other purposes poorly. Though this is no real failing, it does leave a real need.
Since the procedure (a fact forum) is intended to summarize facts, each side will begin by stating what it sees as the key facts and listing them in order of importance. Discussion will begin with the statements that head each side's list. Through rounds of argument, cross examination, and negotiation the referee will seek agreed-upon statements. Where disagreements remain, a technical panel will then write opinions, outlining what seems to be known and what still seems uncertain. The output of the fact forum will include background arguments, statements of agreement, and the panel's opinions. It might resemble a set of journal articles capped by a concise review article - one limited to factual statements, free of recommendations for policy.
This procedure must differ from that of a court in various ways. For example, the technical panel - the forum's "jury" - must be technically competent. Bias might lead a panel to misjudge facts, but technical incompetence would do equal harm. For this reason, the "jury" of a fact forum must be selected in a way that might be dangerous if allowed in courts of law. Since courts wield the power of the police, we use juries selected from the people as a whole to guard our liberty. This forces the government to seek approval from a group of citizens before it punishes someone, thus tying the government's actions to community standards. A fact forum, however, will neither punish people nor make public policy. The public will be free to watch the process and decide whether to believe its results. This will give people control enough.
Still, to make a fact forum fair and effective, we will need a good panel-selection procedure. Technical panels will correspond roughly to the expert committees appointed by governments or to the referees appointed by journals. To ensure fairness, a panel must be selected not by a committee, a politician, or a bureaucrat, but by a process that involves the consent of both sides in the dispute. In court proceedings, advocates can challenge and reject any jurors who seem biased; we can use a similar process in selecting the panel for a fact forum.
Experts who are directly involved in a dispute can't serve on the panel - they would either bias the panel or split it. The group sponsoring a fact forum must seek panelists who are knowledgeable in related fields. This seems practical because the methods of technical judgment (often based on experiments and calculations) are quite general. Panelists familiar with the fundamentals of a field will be able to judge the detailed arguments made by each side's specialists.
Other parts of the fact forum will also resemble those of courts and journals. A committee like a journal's editorial group will nominate a referee and panelists for a dispute. Advocates for each side, like authors or attorneys, will assemble and present the strongest case they can.
Despite these similarities, a fact forum will differ from a court: It will focus on technical questions. It will suggest no actions. It will lack government power. It will follow technical rules of evidence and argument. It will differ in endless details of tone and procedure. The analogy with a court is just that - an analogy, a source for ideas.
A fact forum will also differ from a journal: It will move as fast as mail, meetings, and electronic messages permit, rather than delaying exchanges by many months, as in typical journal publication. It will be convened around an issue, rather than being established to cover a scientific field. It will summarize knowledge to aid decisions, rather than serving as a primary source of data for the scientific community. Although a series of fact forums won't replace a journal, they will help us find and publicize facts that could save our lives.
Dr. Arthur Kantrowitz (a member of the National Academy of Sciences who is accomplished in fields ranging from medical technology to high-power lasers) originated the concept I have just outlined. He at first called it a "board of technical inquiry." Journalists promptly dubbed it a "science court." I have called it a "fact forum"; I will reserve the term "science court" for a fact forum used (or proposed) as a government institution. Proposals for due process in technical disputes are still in flux; different discussions use different terms.
Dr. Kantrowitz's concern with due process arose out of the U.S. decision to build giant rockets to reach the Moon in one great leap; he, backed by the findings of an expert committee, had recommended that NASA use several smaller rockets to carry components into a low orbit, then plug them together to build a vehicle to reach the Moon. This approach promised to save billions of dollars and develop useful space-construction capabilities as well. No one answered his arguments, yet he failed to win his case. Minds were set, politicians were committed, the report was locked in a White House safe, and the debate was closed. The technical facts were quietly suppressed in the interests of those who wanted to build a new generation of giant rockets.
This showed a grave flaw in our institutions - one that persists, wasting our money and increasing the risk of a disastrous error. Dr. Kantrowitz soon reached the now obvious conclusion: we need due process institutions for airing technical controversies.
Dr. Kantrowitz pursued this goal (in its science-court form) through discussions, writings, studies, and conferences. He won endorsements for the science court idea from Ford, Carter, and Reagan - as candidates. As Presidents, they did nothing, though a presidential advisory task force during the Ford administration did detail a proposed procedure.
Still, progress has been made. Although I have used the future tense in describing the fact forum, experiments have begun. But before describing a path to due process, it makes sense to consider some of the objections.
These critics propose no alternatives, and they seldom argue that we have due process today, or that due process is worthless. We must deal with complex, technical issues on which millions of lives depend; dare we leave these issues to secretive committees, sluggish journals, media battles, and the technical judgment of politicians? If we distrust experts, should we accept the judgment of secretive committees appointed in secret, or demand a more open process? Finally, can we with our present system cope with a global technology race in nanotechnology and artificial intelligence?
Open, due-process institutions seem vital. By letting all sides participate, they will harness the energy of conflict to a search for the facts. By limiting experts to describing the facts, they will help us cope with technology without surrendering our decisions to technocrats. Individuals, companies, and elected officials will keep full control of policy; technical experts will still be able to recommend policies through other channels.
How can we distinguish facts from values? Karl Popper's standard seems useful: a statement is factual (whether true or false) if an experiment or observation could in principle disprove it. To some people, the idea of examining facts without considering values suggests the idea of making policy without considering values. This would be absurd: by their very nature, policy decisions will always involve both facts and values. Cause and effect are matters of fact, telling us what is possible. But policy also involves our values, our motives for action. Without accurate facts, we won't get the results we seek, but without values - without desires and preferences - we wouldn't seek anything in the first place. A process that uncovers facts can help people choose policies that will serve their values.
Critics have worried that a science court will (in effect) declare the Earth to be flat, and then ignore an Aristarchus of Samos or a Magellan when he finds evidence to the contrary. Errors, bias, and imperfect knowledge will surely cause some memorable mistakes. But members of a technical panel need not claim a bogus certainty. They can instead describe our knowledge and outline our ignorance, sometimes stating that we simply don't know, or that present evidence gives only a rough idea of the facts. This way, they will protect their reputations for good judgment. When new evidence arrives, a question can he reopened; ideas need no protection from double jeopardy.
If fact forums become popular and respected, they will gain influence. Their success will then make them harder to abuse: many competing groups will sponsor them, and a group that abuses the procedure will tend to gain a bad reputation and be ignored. No single sponsoring group will be able to obscure the facts about an important issue, if fact forums gain reliability through redundancy.
No institution will be able to eliminate corruption and error, but fact forums will be guided, however imperfectly, by an improved standard for the conduct of public debate; they will have to fall a long way before becoming worse than the system we have today. The basic case for fact forums is that (1) due process is the right approach to try, and (2) we will do better if we try than if we don't.
If finding the facts about technology really is crucial to our survival, and if due process really is the key to finding the facts, then what can we do about it? We needn't begin with perfect procedures; we can begin with informal attempts to improve on the procedures we have. We can then evolve better procedures by varying our methods and selecting those that work best. Due process is a matter of degree.
Existing institutions could move toward due process by modifying some of their rules and traditions. For example, government agencies could regularly consult opposing sides before appointing the members of an expert committee. They could guarantee each side the right to present evidence, examine evidence, and cross-examine experts. They could open their proceedings to observers. Each of these steps would strengthen due process, changing Star Chamber proceedings into institutions more worthy of respect.
The public benefits of due process won't necessarily make it popular among the groups being asked to change, however. We haven't heard the thunder of interest groups rushing to test their claims, nor the cries of joy from committees as they throw open their doors and submit to the discipline of due process. Nor have we heard reports of politicians renouncing the use of spurious facts to hide the political basis of their decisions.
Yet three U.S. presidential candidates did endorse science courts. The Committee of Scientific Society Presidents, which includes twenty-eight of the leading scientific societies in the U.S., also endorsed the idea. The U.S. Department of Energy used a "science court-like procedure" to evaluate competing fusion-power proposals, and declared it efficient and useful. Dr. John C. Bailar of the National Cancer Institute, after failing to make medical organizations recognize the dangers of X rays and reduce their use in mass screening, proposed holding a science court on the subject. His opponents then backed down and changed their policies - apparently, the mere threat of due process is already saving lives. Nevertheless, the old ways continue almost unchallenged.
Why is this? In part, because knowledge is power, and hence jealously guarded. In part, because powerful groups can readily imagine how due process would inconvenience them. In part, because an effort to improve problem-solving methods lacks the drama of a campaign to fight problems directly; a thousand activists bail out the ship of state for every one who tries to plug the holes in its hull.
Governments may yet act to establish science courts, and any steps they may take toward due process merit support. Yet it is reasonable to fear government sponsorship of science courts: centralized power tends to beget lumbering monsters. A central "Science Court Agency" might work well, do little visible harm, and yet impose a great hidden cost: its very existence (and the lack of competition) might block evolutionary improvements.
Other paths lie open. Fact forums will he able to exert influence without help from special legal powers. To have a powerful effect, their results only need to be more credible than the assertions of any particular person, committee, corporation, or interest group. A well-run fact forum will carry credibility in its very structure. Such forums could be sponsored by universities.
In fact, Dr. Kantrowitz recently conducted an experimental procedure at the University of California at Berkeley. It centered around public disputes between geneticist Beverly Paigen and biochemist William Havender regarding birth defects and genetic hazards at the Love Canal chemical dump site. They served as advocates; graduate students served as a technical panel. The procedure involved meetings spread over several weeks, chiefly spent in discussing areas of agreement and disagreement; it wound up with several sessions of public cross-examination before the panel. Both the advocates and the panel agreed to eleven statements of fact, and they clarified their remaining disagreements and uncertainties.
Arthur Kantrowitz and Roger Masters note that "in contrast to the difficulties experienced in the many attempts to implement a science court under government auspices, encouraging results were obtained in the first serious attempt... in a university setting." They remark that the traditions and resources of universities make them natural settings for such efforts. They plan more such experiments.
This shows a decentralized way to develop due-process institutions, one that will let us outflank existing bureaucracies and entrenched interests. In doing this, we can build on established principles and test variations in the best evolutionary tradition.
Leaders concerned with a thousand different issues - even leaders on opposing sides of an issue - can join in support of due process. In Getting to Yes, Roger Fisher and William Ury of the Harvard Negotiation Project point out that opponents tend to favor impartial arbitration when each side believes itself right. Both sides expect to win, hence both agree to participate. Fact forums should attract honest advocates and repel charlatans.
As due process becomes standard, advocates with sound cases will gain an advantage even if their opponents refuse to cooperate - "If they won't defend their case in public, why listen to them?" Further, many disputes will be resolved (or avoided) without the trouble of actual proceedings: the prospect of a refereed public debate will encourage advocates to check their facts before they take stands. Establishing fact forums could easily do more for a sound cause than would the recruitment of a million supporters.
Yet today well-meaning leaders may feel forced to exaggerate their cases simply to be heard above the roar of their opponents' press releases. Should they regretfully oppose the discipline of fact forums? Surely not; due process can heal the social illness that forced them away from the facts in the first place. By making their points in a fact forum, they can recover the self-respect that comes with intellectual honesty.
Unearthing the truth may undermine a cherished position, but this cannot harm the interests of a true public-interest group. If one must move a bit to make room for the truth, so what? Great leaders have shifted positions for worse reasons, and due process will make opponents shift too.
Gregory Bateson once stated that "no organism can afford to be conscious of matters with which it could deal at unconscious levels." In the organism of a democracy, the conscious level roughly corresponds to debate in the mass media. The unconscious levels consist of whatever processes ordinarily work well enough without a public hue and cry. In the media, as in human consciousness, one concern tends to drive out another. This is what makes conscious attention so scarce and precious. Our society needs to identify the facts of its situation more swiftly and reliably, with fewer distracting feuds in the media. This will free public debate for its proper task - judging procedures for finding facts, deciding what we want, and helping us choose a path toward a world worth living in.
As change quickens, our need grows. To judge the risk of a replicator or the feasibility of an active shield, we will need better ways to debate the facts. Purely social inventions like the fact forum will help, but new social technologies based on computers also hold promise. They, too, will extend due process.