"The horror of that moment, " the King went on, "I shall never,
never forget!"
"You will, though, " the Queen said, "if you
don't make a memorandum of it."
Lewis Carroll Through the
Looking-glass
Historical Development
In Cabinet No. 1 of the Musee des Antiquites Nationales near Paris there lies a wing-bone of an eagle, not much longer than a finger. On it, three rows of tiny marks, each carefully engraved with a flint point, count off a calendar of days from new moon to new moon. That eagle bone from the Magdalenian period, roughly 14,000 years ago, is the most ancient evidence we have of man's unique ability use abstract notation as an aid to memory.
Out of the Stone Age, through the dawn of agriculture, similar records in all pre-literate cultures attest to the attempts of hunters, gatherers, and farmers to keep track of the passing of the seasons and the meshing cycles of growth and harvest on which survival depended. Even long after more complex societies had fostered more elaborate forms of written record keeping, simple tally scratches, half practical and half magical, continued to serve as records-on the tally sticks of millers, for example, and on the six-guns of lawmen.
Record-keeping techniques grew and were perfected as once scattered tribes and small communities were amalgamated into larger and more organized states. Among the ancient cradles of civilization-Asia Minor, China, India, and Central and South America-only the Inca civilization of the Andes did not develop a written method of recording, using instead a system of knotted cords, called quipu. Indeed, practically all the earliest writing deals with records-palace inventories, lists of tribute to kings and sacrifices to temples, records of royal births and deaths, traders' accounts-records of things too important to trust to memory.
In most of the ancient world, the scribes and clerks who developed systematic record keeping quickly expanded into generalized public administration. In Sumer and other city-states of Mesopotamia, royal genealogies were embellished with accounts of battles, land. surveys included detailed descriptions of farms and villages, and tax records included commentaries on the tax laws that governed them. Gradually these commentaries were detached from records proper and took on a separate existence. The law code of Hammurabi, for example, emerged from the notes of scribes and marks an important milestone in the history of social organization. Once the laws of the state achieved an existence independent of records, the witness of the records could be used to bind the state and the citizen equally. When both the tax laws and the size of a man's herd were matters of public record, the pressure of public scrutiny would tend to keep both the publican and the herdsman honest.
Systematic record keeping in the ancient world reached a high point during the Roman Empire and then degenerated with the decline of strong central government. During the Middle Ages the levying of taxes was left largely in the hands of local strongmen who had little interest in record keeping. Although the laws of inheritance and the interest of the Church in proper sacramental procedures encouraged parishes to maintain registers of births, marriages, and deaths, those records seldom covered the bunk of the population. In soma cases, however, rulers of newly conquered domains did order inventories and land surveys. One such was William the Conqueror's survey, known as the Domesday Book, of the extent and value of landholdings in England in 1086 A.D.. It became the foundation of Exchequer records that, in turn, grew to include audits of the accounts of sheriffs and other local officials. The memory-aiding function of these records is suggested by the title of the official responsible for keeping them-King's Remembrancer.
As a landmark in the gradual evolution from personal sovereignty to bureaucratic administration, the Magna Carta of A.D. 1215 laid the foundation in Anglo-Saxon legal tradition for codifying mutual responsibilities of government and governed. The Magna Carta, wrested from King John by his powerful barons, reduced the independence of justices, sheriffs, and other local officials, censuring, in theory at least, that men who knew the common law and were willing to observe it would hold positions of high authority. During the reign of King John also, an administrative distinction between public and closed records began to be observed; official records were divided into letters patent that were sent and stored open, with the king's seal attached for authentication, and letters close that were sent folded and sealed, and that were stored secure from public inspection. The use and content of these two classes of records corresponds well, as we shall see, with the modern practice of separating public from confidential records.
As custom and statute more and more provided that government records should be open to the public, the justification for closed or secret records came to be their pertinence to the defense and security of the state. By the mid-1600's; all royal courts maintained files1 of information on the identity and activities of citizens or aliens who were considered a threat to the state or the sovereign. Such files covered a small number of individuals by today's standards, but were treated with great secrecy and came to be the responsibility of a special class of record keeper well outside the regular channels of administration. The scope and intensity of this special field of record keeping soon gave it a character so different from its bureaucratic origins that it becomes convenient at this point to draw a distinction between general administrative records and the very special intelligence records.
As the idea gradually spread that governing a state involved more than determining and following the wishes of a small ruling class, government became less desultory, more aligned to philosophical currents, and less reactive to the press of random events. As government thus grew more self-conscious, the need for planning became apparent. At first, legislators used their right of access to public records mainly to look backwards; to reconstruct the flow of history that had brought them to their present position. However, lawmakers bent on reform soon found that they needed better guides than records of legal decisions, royal correspondence, and official accounts and audits. They needed benchmark information from which to measure progress toward the goals they wish to achieve.
About 1750, the notion of a national census was revived for the first time since the Roman era. Public opposition was strong at first, many people suspecting a scheme to raise taxes. The clergy, for whom the Biblical injunction against the taking of a census still held,2 also were opposed. Resistance gradually subsided.; first in Scandinavia and the German states, then generally throughout the Continent and North America. In the American democracy, where a State's Congressional representation constitutionally depends in part on the size of its population, a national census, at ]east to the extent of a simple head count, was an obvious political necessity.
Government soon found that although there was little organized public objection to the head count as such, probing by census takers for information about income, family life, living habits, and other personal matters turned citizens obstinate and made the census more difficult to take.
The problem of gathering information from an antagonistic public led to the creation of yet another class of official. records, the so-called statistical3 file. The essence of such a file is that the data it contains are not used to affect specific individuals. In creating such a file, the government, in, order to gain information the public might otherwise be reluctant to give, foregoes some of the power over individuals that administrative records containing the same data would afford. The essential condition is that citizens believe that their individual contributions to a statistical file will not be made public and will not be used to punish or embarrass them.
Types of Records About People
As we approach the computer age in this brief survey of record keeping, we need to define the three main types of records that have been distinguished historically4.
Administrative Records. The administrative record is often generated in the process of a transaction-marriage, graduation, obtaining a license or permit, buying on credit, or investing money. Usually a record that refers to an individual includes an address or other data sufficient for identification. Personal data- in an administrative record tends to be self-reported of gathered through open inspection of the subject's ,affairs. Private firms usually treat administrative records pertaining to individuals as proprietary information, while administrative records held by the government are normally accessible to the public and may be shared for administrative purposes among various agencies. Administrative records sometimes serve as credentials for an individual; birth certificates, naturalization papers, bank records, and diplomas all serve to define a person's status.
Intelligence Records. The intelligence record may take a variety of forms. Familiar examples are the security clearance file, the police investigative file, and the consumer credit report. Some of the information in an intelligence record may be drawn from administrative records, but :much of it is the testimony of informants and the observations of investigators. Intelligence records tend to circulate among intelligence-gathering organizations and to be shared selectively with organizations that make administrative determinations about individuals. Intelligence records are seldom deliberately made public, except as evidence in legal proceedings.
Statistical Records. A statistical record is typically created in a population census or sample survey. The data in it are usually gathered through a questionnaire, or by some other method designed to assure the comparability of individual responses. In nearly all cases, the identity of the record subject is eventually separated from the data in the record. If a survey must follow a given individual for a long time, his identity is often encoded, with the key to the code entrusted to a separate record to guard anonymity. Data from administrative records are sometimes used for statistical purposes, but statistical records about identifiable individuals are generally not used for administrative or intelligence purposes.
Not every record falls clearly into one of these three categories. The contemporary personnel record combines features of bath administrative and intelligence records, and the records in the modern "management information system" have both administrative and statistical uses. Many records share characteristics of all three types to some degree. Yet whether one looks at the relationships among records of different types historically, from the perspective of present-day public policy, or from the point of view of the individuals who are the subjects of records, it is apparent that, by and large, administrative records are considered public; intelligence records, secret; and statistical records, anonymous. Moreover, democratic traditions with respect to the maintenance of government records about people have deep historical roots in a number of countries,5 and appear to be dominated by three major principles.
From Record Keeping to Data Processing
In this country, the end of World War II unleashed the deferred wants and pent-up purchasing power of the war years onto a labor-poor, capital-rich market. To help deal with the social and economic dislocations created, first, by demobilization, and later, by the Cold War, government kept in force many of the controls it had established during the years of all-out mobilization. The nation's pride in its wartime accomplishments lent a tone of confidence to even the most ambitious planning. Industry, for its part, took advantage of new technologies emerging from. wartime research and development to make revolutionary changes in its methods of producing and distributing goods and services.
Acting together, these forces rapidly expanded commercial and governmental activities in the late forties and early fifties, forcing a vast increase in the volume of transactions requiring records about people. Compared with pre-war years, the number of bark checks written, the number of college students, and the number of pieces of mail all nearly doubled; the number of income-tax returns quadrupled; and the number of Social Security payments increased by a factor of more than 35.7
Technology developed during the war years was available to meet the challenge posed by this rising tide of recorded transactions. Automated data processing, transplanted from its military origin, quickly blossomed into a powerful industry, feeding on the demands of commerce and government for fast and efficient data handling, and in turn, fueling that demand by significantly changing the philosophy and practice of management itself. Since most industries based on a highly technical product must quickly develop a mass market to recover the high development and tooling costs, the computer industry devoted much attention and talent to marketing its products, without appreciating the implications of the technological revolution it was unleashing.
By the 1960's, attractive prices, persuasive salesmen, and ingenious computer software services had stimulated the introduction of automated data processing equipment into a great many record-keeping organizations, sometimes with far too little attention to the objectives and costs of automation. Although there were many examples of diseconomies and a few outright failures, the successes were so spectacular that the prestige of having a large-scale data processing capacity often prompted managers to keep their computers running, even at a financial loss.
The computer scored its earliest successes as a record keeper in fields where the data were mainly numerical. The speed with which the computer can do complex arithmetic, and the compactness of numerical data as compared to natural language, were major factors in quickly amortizing the considerable expense of installing a computer, and of converting an established record-keeping operation to take full advantage of the computer's capabilities. Thus, the earliest successes were heavily concentrated in science and engineering, banking, insurance, and accounting, and, above all, in the space program, where the value of computers in handling the intricate logistics of production, assembly, and testing was soon discovered.
Systematic Management
For computers to be used effectively as management tools, an organization must first analyze its activities in a careful, systematic way. For example, if it is known that the goals of an operation can be attained by more than one method, the various alternatives can in principle be simulated on the computer, and their relative costs and benefits thereby compared to find the most cost-effective one. This mathematical simulation of a complex activity is called systems analysis.
During the late sixties, planners began to extend the techniques of systems analysis from their early engineering applications to more general problems of society. In particular, systems analysis was brought to bear on such ambitious tasks as improving the delivery of health care, managing the rapidly growing welfare caseload in urban centers, and measuring the effectiveness of a fragmented and increasingly expensive educational system.
The introduction of the disciplined methods of computer-assisted management gave program managers new tools for "auditing" the performance of institutions in programs of service to people. This auditing process includes:
Each of these functions involves information about individuals. Administrative data are needed for everyday management of individual transactions. Statistical data are needed for planning and for assessing the performance of a program. Intelligence: data are needed for making judgments about people's character and qualifications; e.g., in making suitability determinations for employment, commercial credit, welfare assistance, tuition-loan aid, or disaster relief.
The demand generated by all these uses for personal data, and for record-keeping systems to store and process them, challenges conventional legal and social controls on organizational record keeping. Records about people are becoming both more ubiquitous and more important in everyday life. The number of organizations performing service and control functions is growing. In many cases, the scale of their operations virtually assures that the individuals they affect will be known to them only through the contents of systematically maintained records. A new technology is also demonstrating its potential to accommodate radical growth in organizational record-keeping operations. Yet society currently affords little protection for an individual who is the subject of a record, unless some commercial or property interest in involved.
The following chapters represent our effort to demonstrate why this situation deserves immediate attention and to recommend a course of action that, we believe, constitutes an appropriate societal response to the problems at hand.
1 The use of the word "file" in this sense dates from the 1640's. See "File," Oxford English Dictionary, 1933,1V, 210.
2II Samuel 24 and I Chronicles, 21, 23, 27.
3The word statistics [state-istics] came into use in the late 18th century to denote information on the condition of a state. See "Statistics," Oxford English Dictionary, 1933, X, E64.
4The classification follows that of Prof. Alan Westin in M. Greenberger (Ed.), Computers, Communications, and the Public Interest (Baltimore, Md.: The Johns Hopkins Press), 1971, p. 156.
5The reader who is interested in comparing the American experience with that of other nations will find a summary of available material in Appendix B, below.
6The evolution of Federal policy with respect to the confidentiality of census data is traced in Appendix C, below. See also Daniel J. Boorstin, The Americans: The Democratic Experience (New York: Random House), 1973, Chapters 19-28.
7Alan F. Westin, and Michael A. Baker, Databanks in a Free Society (New York: Quadrangle Books), 1972, pp. 224-225.