Legal Implications of Complex Virtual Reality Systems Speech delivered to DEFCON I (Las Vegas, July 10, 1993). Curtis E.A. Karnow1 I. INTRODUCTION A. We know about criminal problems 1. New Crimes--and old fashion crimes--using computers.2 a. breaking and entering b. trashing information: viruses C. changing information d. Copyright - the new federal felony e. The scope of the federal government's interests: "federal interest" Computers: any communications across state lines triggers the interest of federal law enforcement. B. Now, as the industry matures, we face civil litigation issues. 1 Curtis Karnow is a partner at the San Francisco law firm of Landels, Ripley & Diamond, and chairs the firm's Competitive Practices Group. His practice emphasizes intellectual property litigation and computer law. He is a faculty member with the American Arbitration Association, a former federal prosecutor, and serves as temporary judge with various Bay Area courts. He can be reached through the Internet: karnow@cup.portal.com. These materials are copyrighted (c) 1993 by Curtis Karnow. The reader's professional courtesy is appreciated in providing attribution to the further use of these materials. 2 See generally, "Note, Computer-Related Crimes," 30 Am.Crim.L.Rev. 495 (1993). 2 II. BACKGROUND: WHAT'S NOW AND WHAT'S COMING IN VR A. What is VR By 'virtual reality' I mean massively networked, broadband, interactive immersive computer-mediated experience. The data, including audio visual displays, feedback and other sensory input are a function of the operating system, multiple layers of application software including those running on a host or hosts, together with remote databases and programs resident on remote platforms. New Techniques 1. direct retinal imagery 2. VDT cursors controlled by amplified brainwaves 3. the surround: a. Battletech b. SEGA goggles c. Sega "Activator" infrared beams for the body movement d. the CAVE B. We're moving from single actor to multiuser 1. i.e. from one communicating simply with a CPU and software to an environment in which many people, from around the world, are contributing in real time. 2. MUDs 3. Given the choice, people generally chose to interact with people, not machines. The experience of game publishers reveals this: think of the attraction of the Head-to-head modes for flight simulators like Falcon 3.0; like Sierra online. C. We're moving from an exoskeleton to the elimination of the interface. 1. From the goggleheaded, bodysuited, multiwired oddball to the user who sheds his chains, his body, and walks into the new universe 2. You know about the Zero insertion force socketZIF? You know GUl? Here's a new three letter word- ZUI- zero user interface. That's what we're aiming at. We want to for et about the technology; we don't want to see it, hear, it, smell, it or 3 face it; we want to think it dissolved, gone, vanished, we want to take it completely for granted, until we believe the environment is real. Computers will dissolve too. When we talk about totally immersive VR, or chips lodged in the brain, or other metaphors for complete integration between mind and machine- that's what we getting at: ZUI [zoo-ee] zero user interface; the vanished machine. 3. What is it for the environment to be real? You think this room is real? The street outside? A person from the 14th century would't be sure, and someone from the Stone age would be sure- that this was a dream. Things get real, after a while, after we adjust to them, and after we go onto other things, think about new issues, and start using the technology for other goals. Like the zipper, the car and phone: these things disappear on you as you use them, transparently, for other work. I don't worry about whether the voice I hear on the phone is "really" my client or friend- the sound I hear is vaguely connected, vaguely relates, to the "real" voice. I don't mind. I think it really is the voice of my friend. Later, I will discuss our willing suspension of disbelief, and our extraordinary ability to feel comfortable in initially extraordinary circumstances. Let me simply note here that a "real" environment is just one that we know well, that responds at such a speed and sufficiently predictable that we ignore it, that we take it for granted.3 3 So I reject attempts to distinguish synthetic from natural environments. All environments are equally synthetic or natural, depending on who or what is doing the sensing. See, 25 Siggrahic Computer Graphics No.3 as an appendix to a report to the 1992 NSF Invitational Workshop "Research Directions in Virtual Environments," later expanded by Warren Robinett in the article "Synthetic Experience: A Proposed Taxonomy." That "taxonomy" suggests synthetic reality or "Synthetic Experience" as opposed to a "Natural Experience". The latter requires the direct perception of some properties or behaviors of something physically present before the perceiver. A "Synthetic Experience" is the perception of a representation or simulacrum. 4 D. Interim Conclusion: Mulituser + ZUI ¯ real interaction with real people and with real parts of the real world. Now, you think the world of courts, lawyers and judges will step in? You bet. Murphy: If something can go wrong, it will. Karnow: If something goes wrong, someone will sue. III. VR AFFECTED BY THE LAW A. Abstract: the Complex Backdrop Summary: Error as a function of at least four general problems. First, the high complexity of computer code; second, the theoretically inability ever to completely debug code; thirdly, the fact that VR systems rely on hidden interpolation, and may utilize fuzzy logic; fourthly, the extremely high speed of useful projected VR systems. 1. High complexity in computer code The software to be used in high end VR systems is, and will increasingly be, complex. This complexity is a function of at least four factors: the size of the programs, multiple providers of software and hardware; multiple simultaneous users, and the intentional programming and anticipation of chaotic behavior. a. New Operating systems such as Pink: extremely large code. Microsoft NT is reputed to exceed 12.5 million lines of code. Add applications, driver software. Contrast: Lines of code Handheld bar code scanners 40000 Electronic four speed transmission 19000 Cellular telephone 30,000 Air traffic controller computer 130,899 Automated Teller network 600,000 Telephone router 2,100,000 New Operating System (Pink) 3,500,000 (could be more:) 5 B-2 bomber 3,500,000 Seawolf Submarine combat Computer 3,600,000 b. Multiple entities: software developers and publishers, telecommunications nets [fibre optic, satellite, etc.] , hardware platform vendors and operators. No one is in charge. c. Many simultaneous users. d. High complexity and unpredictability will be sought out by designers to mimic "real life". Objects must have many properties. High chaos is engendered by interacting autonomous computational agents- the most complex of which are interacting human beings!4 Virtual reality's power lies in its wholly immersive, kinaesthetic front-end: and importantly, the kinaesthetic interface will be a function of many users: both the users currently in the environment and probably prior users. A truly sophisticated v.r. environment will be extremely complex: it will be theoretically unpredictable; that is, chaotic, just as real complex systems exhibit rule-bound, but chaotic behavior. Indeed, I suggest it is this unpredictability, these open-ended processes, that will make a crucial contribution to the felt perception that the experience qualify as a convincing virtual reality. 5 Current personal [stand-alone] computers are relatively simple, because they are basically uncoupled from the user and the rest of the environment; or, more accurately, they are coupled in a few simple and direct ways: data input and output- keyboards and printers, screens, and such, and some dependence on the stability of the electrical/hardware system. But: as the number of elements of the systems increases, as the number of the semi-autonomous agents -- humans and computer - 4 See C. Morningstar & F. Randall Farmer, "The Lessons of Lucasfilm's Habitat," Cyberspace: First Steps (M. Benedikt, ed. 1992), reprinted in Virtual Reality 93 - Special Report (Al Expert) 23, 25. 5 See C. Morningstar & F. Randall Farmer, "The Lessons of Lucasfilm's Habitat," Cyberspace: First Steps (M. Benedikt, ed. 1992) , re rinted in Virtual Reality 93 - Special Report (Al Expert) 23, 25. 6 - increases, chaotic, unpredictable behavior will emerge. Recall that chaotic evolution can be generated with only three interacting independent systems.6 2. Theoretically Inability to debug a. Millions of lines of code interacting with each other and in the "real world" environment. Studies, including some published in Scientific American, establish that moderately complex programs cannot be completely debugged, and that probably no program can be thoroughly debugged. For the mathematicians among you, the issues are related to the halting problem: there is no way to determine when all bugs have been eliminated. There is no way to simply examine a program and tell if it is buggy; instead, the program must be actually executed, and run under all possible circumstances, to detect many bugs. And, of course, there is not enough time in the life of the universe to do that for most useful programs! b. Think of the complexity of developing a single bulletproof applications package: "handling memory and system kludges, BIOS incompatibilities, DMA and IRQ conflict madness, funky foreign motherboards, different DOS and Windows and OS/2 versions, different motherboards, different address schemes, different drive adapters, different keyboards, different busses, different compression schemes and even different mice. . . ."7 3. Interpolation and Fuzziness The synthetic appearance of virtual worlds, to be "realistic" must be the result, at least in part, of interpolation and of fuzzy processing. Blanks are not an acceptable part of the illusion: the software must fill in the blanks, and must allow for spectrums of colours, sizes and shapes and other dimensions of objects. When you recall that (1) all objects and their properties or dimensions are, in fact, data, and (2) users are making decisions in a virtual world based on the properties of the objects presented, then it becomes clear that inaccuracies of perception, and then inaccuracies in user conclusions, are possible-- not as a result of error in the classic sense, i.e. not as a result of bugs or glitches or unforeseen effect of multiple inputs, but as a direct function of the calculation at issue, as a direct result of the fact that the problem is fuzzy, or the software has interpolated likely values for the data. 6 D. Ruelle, Chance and Chaos 81 (1991). 7 "Dvorak V. Somerson" PC Com utin jan 1993 at p. 108 7 We handle this all the time in real life; but in a virtual world we may forget, or be unaware, that this is going on. For example, we examine a chart showing stock performance, and associated facts making up the historical performance of the company, the economy, and so on. Looking at a paper chart, IF WE THINK ABOUT IT, we can tell where the data in incomplete, where interpolating lines join known data points, where two end points create a range of possible values. Now consider a virtual world, where the user flies through mutlicoloured and mutlishaped terrain, which provides the information to him at a virtual glance. This application currently exists. Under these circumstances, it can be difficult to separate out plainly what is known and what is not. The goals and techniques of virtual reality development make this problem of interpolation especially problematic. Recall that the success of the computer-generated immersive experience depends on our willing suspension of disbelief: that is, out ability to pretend, and to believe that pretence deeply. When we enter a virtual world, our point of view changes dramatically. We might be looking out the eyes of a robot across the room, looking back at our own body: We will believe that we really are out-of- body, looking at ourselves. We might be flying above terrain, down blood vessels, across coloured charts, or sitting in a room talking to people whose real bodies are in Iceland. In a successful VR environment, we will act as if we are in those places. Experiments have shown that even when one sense- such as vision- is replaced by tactile feedback such as patters of tingling on the skin, subjects will react as if they really see- such as lurching backwards when the tingling sensation is artificially amplified.8 In short, a shift in the felt location of consciousness can be induced by a variety of means, as every good designer of exciting, interactive games intuitively knows. The sense of self and of consciousness is identified generally with the body, of course, but only when our attention to drawn to the physical body, and only in default of some other potential location for the reception of sensory experience. As Marvin Minsky has noted, 8 Daniel C. Dennett, Consciousness Ex lained 341 (1991) . 8 We have the sense of actuality when every question asked of our visual systems is answered so swiftly that it seems as though those answers were already there.9 Powerful virtual reality routines of necessity will play on our enormous capacity for belief, to make sense of, and find patterns in, whatever surrounds us, especially with respect to the basic propositions of where and what we are. In this context, it may be impossible to separate fact from fiction, data from interpolation, and substance from cosmetics. 4. Performance: Very high speed. The illusion of reality depends on it. We want no latency: that's how we measure gratifying power in today's systems. We want, and we will get, immediate results, immediate effects. Instant Sensual Gratification! So, the sum of these parts: High Complexity + High Speed + Interpolation and fuzzy data + Inevitable Error + ¯ Potential civil liability ¯ as ZUI becomes the norm. B. Forgetting Invisible Technology To some extent, visible, kludgy technology is its own warning; it contains its own seat belts. You can't go too fast in a model T Ford, you don't get too many accidents. Visibly problematic technology has a built-in safety belt: people are careful as they manipulate the machine; they don't rely without question on the product; they watch for the final result, and have, it follows, an independent but simultaneous sense of when the technology is working. Later, as the technology gets integrated, works well, and finally disappears from our consciousness, the checks and balances evaporate. At first, we maintain an independent sense of whether the result is right, that something is working, but that isn't simultaneous: we check later, and maybe we find the error and maybe we don;t. Current spreadsheet applications fall into this Category, as does aircraft manufacturing: we're pretty sure it 9 The Societ Of Mind 257 (1985) . 9 works, but can't check in real time. When the numbers looks bad in retrospect, or engines fall off the wings of a 747, the damage is done, but we do go back and fix it. Later, we loose the ability to cross-check the results,10 or fix them. Voice mail and phone message machine? This is most apparent in highly complex computer systems, and in complex economic and social systems. The reason is, in brief, that the systems are sufficiently complex that there is no such thing as a control group; the system cannot be literally duplicated. For example: when one first begins to use a calculator, one has a sense of when the result is right; and one might check the calculator by hand- my wife does! When it works repeatedly, one considers using the technology for calculations that cannot be otherwise checked: there is no independent, simultaneous sense of whether the technology works. We assume it does, like we assume the phones will work and a meteor will not strike the earth, and we get on with our ultimate tasks without worrying about the tools. In a sophisticated high-speed ZUI VR environment, we may not know if and when the system crashes - or not until it's too late to avoid injury and death. How real a problem is that likely to be? C. The real world's infrastructure is based on software: it makes the industrial world go round: There is a reasonable probability of injury as VR interacts with real world events. [Indeed there is no VR application that does not interface with the real world, because it always at least interfaces with a real human.] As computers become more advanced and sophisticated-- and VR represents the Cutting edge of that sophistication -- computers 10 Loosing the ability might also mean not having time to do so. Manual searches of databases are possible, but not reasonably feasible. Akins v. District of columbia, 526 A.2d 933 (Dist.Colum. March 31 1987) (criminal justice system computer failed to generate prior reports of convictions, resulting in release of dangerous felon; manual retrieval not attempted) . Loosing the ability may simply mean that a function, which could be done by a human or otherwise cross-checked, has been permanently delegated to a machine, such as data retrieval. Brown v. Unites States, 790 F.2d 199, (1st Cir. 1986) (damaged weather buoy provided erratic data, ultimately leading to death of fishermen who relied on data) . 10 become more reliable and more integrated into our infrastructure; BUT when they fail-- and they will fail-- it is stunningly spectacular. 1. Failure Examples: : a. Medical procedures /telepresence b. recall the last Black Monday on Wall Street, 1987? when computer based trading brought the market down c. Computer bugs / SRI list11 (1)3 bits of code brought down the east coast phone service between June 26 July 2, 1991. (2)1987: lethal doses of x rays from Computer controlled machine; glitch when commands typed in too rapidly- buffer problem. (3)Crashes of Army Blackhawk helicopters in the mid-1989: EMF radiation caused problems in unshielded connections between inflight computer and control surfaces. (4)Confusing interface for the missile launch system on the Aegis cruiser Vincennes contributed or caused the launch of the missile that shot down the civilian Airbus Iran Air flight 655 (290 dead). (5)Error in modeling programs, or inappropriate modeling, responsible for collapse of buildings (I.e. Hartford Civic Center) and other products such as ships. d. Training: SimRefinery; military; NASA e. product defects f. MUDs take a hug amount of Computing power and communications bandwidth. They can and do cripple computer networks. They are illegal on many campuses, and banned in the entire continent of Australia. 2. How do these issues get translated in to the legal world? HOW do legal rules get implicated, how do the courts get involved? If this were a group of lawyers, we could chat about torts, various specific statutes, and the differences between deign defect and manufacturing defects, implied warranties, and the rest of it. But none of that matters, at heart. The law is enormously flexible: ancient doctrines are applied in the twinkling of an 11 Peter Neumann, FTP CRVAX.SRI.COM and go to the appropriate directory with "CD RISKS:" 11 eye to new technologies. The point is that is if anyone thinks they have been wronged, injured, hurt -- if anything in the real world goes not accordingly to plan -- to someone's plan-- then that someone can file a complaint and start a lawsuit. It only takes about $80 or so to pay the filing fee in state court in california. The legal community has, or will have, an interest in the unforeseeable results of complex virtual reality systems. Two highly complex systems are merging- virtual systems and the law. The results of that are -- unpredictable.