Read Transcript ↓
For a General Theory
of Simulations
Everything you do is training data for future model of the past
University of Southern California, Los Angeles, January 2024
Benjamin H. Bratton
So we find ourselves here at the USC School of Cinema, adjacent to the George Lucas and Steven Spielberg buildings. One of the center of the many centered simulation industry, where the craft is both theorized and taught.
So I'll begin with a acknowledgement of sorts that the talk you will see tonight is understands itself to be training data for some future model about this present moment. In fact, everything you do today was training data for some future model of the past. For the Hawthorne effect, we call our shared reality. The present is a simulation of itself as the past. Put differently, moments in time have shadows, as do each of us.
What is a shadow, however? It is something both obvious and mysterious, perhaps one because the other. An object can be seen through the reflection of light, but this reflection is also a filtering so that the light cannot reach whatever is on the other side of that object. This produces an artifact which can be interpreted as both a kind of subtractive absence and as a new kind of thing with significant but incomplete attachment to the original object whose profile it resembles.
This secondary void/object is what is called a shadow. It tracks the object from which it seems to be projected but also changes in shape and size in ways the original object cannot. At times the shadow appears as a dark two-dimensional replica of the object but if, for example, the source of light is closer to the ground plane then the shadow bends and elongates and in doing so seems to take on a formal identity of its own different from the object.
Anyone who spent their childhood watching cartoons can half-remember characters peeling off their own shadow, moving their shadow to a different spot, dancing a duet with their shadow, boxing with their shadow, and so on. The allegories at play here are less metaphysical than psychological. Generally they are similar to the dilemmas posed by a mirror reflection in which self is recognized as an exterior object and specifically they play with a potentially horrifying loss of self-control, or perhaps control over the external effects of one’s thoughts, emotions, and actions. To box with one’s shadow is to be of two-minds about something with existential consequences. To peel off one’s shadow and fold up into a suitcase (as a cartoon cat once did) is to decouple oneself from those consequences of agency, or at least to enact the feeling that such agency is already decoupled.
Speaking of which….Plato’s allegory of the cave is a theory of simulation at the core of Western philosophy and is concerned not only with shadows misperceived as primary objects but with the prospect that what we take to be everyday objects are themselves a kind of shadow of yet more primary forms.
This worry haunts the ongoing critical suspicion of phenomenal appearances and what does and does not constitute the correspondence between image, truth and reality. Perhaps then the philosophy of simulation begins with the beginnings of philosophy itself. They were not just a topic for philosophy but perhaps a kind of foundational paranoia from which philosophy was born.
In my misspent youth, a took a trip bumming around Indonesia, and one of the few things I brought with me was a a dog eared copy of Simulations by Baudrillard. The true object of which I show you here.
Among my favorite memories from this trip attending late night epic dramas told with shadow puppets in open air temples in the Javanese city of Yogyakarta. Crowds of people wandering in and out, would share tall bottles of beer and various snacks as the six, seven, eight hour epic would unfold and cycle back around and start again over and over until dawn. Shadow puppetry may stage a similar dilemma but by inverting Plato’s allegory.
This particular “cave” accessed the agency of the shadow perhaps more directly and presents them not as deceptive illusions but as direct presentations of the fundamental mythic systems that underlie daytime reality but which are otherwise obscured by it.
Part 1: A General Theory of Simulation
What about here and now? Today we live in with within caves of our own making. We build simulations that get us both closer and further from reality. Arguably all formal simulations are based on models, but not all models are simulations. That is, different forms of simulation are based on different sorts of models.
Start with some theory before diving into examples which will be more fun.
Some models are descriptive –their value is in how well they correspond to some reality–, others are primarily predictive - their value is in how well they anticipate a future state condition, and others are projective or normative – their value is in how they articulate a possible or preferred reality that does not now exist and may be unlikely to exist if not for the power of the normative model.
What do I mean by this? Descriptive models are, for example, the domain of simulation science seeking to understand complex natural phenomena; predictive models such as financial simulations hope to guess market futures; and projective or speculative simulations are the domain of, for example, religious prophets, politicians and designers, seeking to give shape to a better world.
That said, it is not always clear which kind of simulation is at work, as it can easily appear to be one when it is in reality another. What claims to be predictive may really be speculative. What claims to be speculative may really just be descriptive. Just ask Oedipus. A prophecy that at first seemed like a speculative simulation turned out to be a predictive simulation and then, tragically, a descriptive one. Setting aside the power of fate, Arguably, it was because he received the prophecy that it came true. The simulation made itself more true because he acted through it. This is an important general principle of recursion.
That is, different kinds of simulation are not only affected by what they simulate in different ways, they also in turn affect what they simulate in different ways. Simulations are not mere representations, they are technologies with feedback built in.
This feedback is understandable through two very important principles for a general theory of simulations: these are reflexivity and recursion.
Recursive simulations are those that directly and automatically affect what they simulate. If there is a change in whatever reality is being simulated, this is automatically updated in the simulation, but also if there is a change in the simulation, this automatically affects the simulated in the same way, whether or not anyone means for it to happen.
Donald MacKenzie famously described financial and economic models in this way, calling them “an engine not a camera.” The computational acceleration of financial simulations and their pervasive deployment makes these feedback loops only more intense and complex. Simulations of the future come to structure action in the present by determining what is most likely and giving it a price.
The simulation of the future comes true because it determines the present that later becomes the future. Insurance and prognosticative simulations of future risk determining what to allow or disallow in the present is but one example.
Recursion can be direct or indirect. It can be a literal sensing/actuation cycle, or an indirect negotiation of interpretation and response. The most nuanced recursions are reflexive. They mobilize action to fulfill or prevent a future that is implied by a simulation.
To be sure, Moore’s Law only accelerated the resolution of simulations, the intricacy of their reflexive and recursive feedback loops and the dizzying complexity of their interaction with each other. The simulations are also simulating each other, predicting what they will each predict about each other's predictions, and on it goes.
Part 1: Scientific Simulations
I want to turn our attention then to a tour though several kinds of simulations, or really several ways in in which those kinds of simulations are used, how each of us participates in them, and indeed how each of us thinks and experiences the world through simulations.
I will begin with how simulations allow us to know the reality of universe.
Scientific simulations not only do more than deceive us, they are, arguably, the essential mechanism by which otherwise inconceivable underlying realities are accessible to thought. From the very very small in the quantum realm to the very very large in the astro-cosmological realm, computational simulations are essential not just as a tool, but as a way of thinking with models, a fundament of induction, deduction and abduction.
Far from hiding reality from us, simulations are essential to how reality is disclosed at all. Stanislaw Lem makes the distinction between what he called instrumental technologies and existential technologies. The former affect the world simply by what they do functionally. The latter, such as a telescope or microscope, when used properly not only let you see at different scales, they change how you understand how the universe works. So all up and down the Standard Model of Physics, clearly simulations, like computation itself, are both an instrumental and existential technology. From planetary science to quantum field theory, from algebraic topology to astrophysics.
At the same time, simulations are based on models of reality, and the status of the model has been a preoccupying concern in philosophy of science, even if simulations as such are more presumed that philosophized. Models are a way of coalescing disparate bits of data into a composite structure whose whole gives shape to its parts, suggests their interactions and general comparisons with other structures. It is a tool to think with.
Its value is in its descriptive correspondence with reality, but this correspondence is determined by its predictive value. If a scientific can predict a phenomenon, its descriptive quality is implied. A model is also, by definition a radical reduction in the variables, like a map reduces a territory. A geocentric or heliocentric model of the solar system can be constructed with styrofoam balls, and one is definitely “less wrong” than the other, but both are infinitely less complex than what they model.
This is especially important when what is simulated is as complex as the universe itself. Astrophysics is based almost entirely on rigorous computational simulations of phenomena that produce difficult to observe data, assembled into computationally expensive models, and which ultimately provide for degrees of confident predictability about astronomic realities that situate us all.
For example, basically all the exoplanets we know of, or planets outside our solar system, have been discovered since the fall of the Berlin Wall. As Anthropologist, Lisa Messeri chronicled, the science of exoplanets is based on constructing model simulations from heterogeneous bits of data, and for the scientists, of understanding what exoplanets are like through imagining them as places, places which are mental simulations of planets many light years away.
This is what we call cosmology, the meta-model of all models of reality in which humans and other intelligences conceive of their place in space-time. Today cosmology in the anthropological sense is achieved through cosmology in the computational sense.
Quite often, though, the simulation comes first. It’s predictive ability may imply that there must be something we should look for because the model suggest it has to be there.
Thus the prediction makes the description possible as much as the other way around.
Such is the case with Black Holes, which were hypothesized and described mathematically long before they were detected let alone observed. For the design of the Black Hole in the Nolan brothers’ film, Interstellar, scientific simulation software was used to give form to the mysterious entity based on consultation with Kip Thorne at CalTech and others. The math had described the physics of black holes, and the math was used to create a computational model that was used to create a dynamic visual simulation of something no one had ever seen.
Of course, a few years later we did see one. The Black hole at the center of the M87 galaxy was “observed” by the Event Horizon telescope and a team at Harvard that included Shep Doelman, Katie Bowman (seen here) and Peter Galison. And, well, it turns out we, the humans were right. Black Holes look like what the math says they must look like. The simulation was a way of inferring what must be true, where to look, how to see it and, only then did the terabytes of data from Event Horizon, finally, a picture.
It is important to note that this conjunction of entertainment purposes with scientific inquiry is by no means anomalous. It is no secret that the general capacity for scientific simulation depends on the availability of cheap GPU’s. GPU’s are cheap because of the economics of video games, an entire genre of entertainment based interaction with computational simulations just for fun. Science is paid for by fun. AI is also driving GPU development, and obviously, will likewise cross both scientific and entertainment domains.
A few years ago, David Krackauer from Santa Fe Institute co-authored a paper on the essential role of simulation across scientific disciplines and proposed an ideal simulation software stack for this purpose. The paper offered Nine Motifs for Simulation Intelligence, a novel way of rethinking the boundaries between scientific disciplines by how they correspond with similar and different forms of simulation. What it didn’t address is that science is but one application for a software/hardware stack that is being driven by other purposes.
Let me emphasize, however, scientific simulation not only has deep epistemological value, it also makes possible the most profound existential reckonings. Climate science is born of the era of planetary computation. Without the planetary sensing mechanisms, satellites, surface and air sensors, ice core samples, all aggregated into models and most importantly the supercomputing simulations of climate past present and future, the scientific image of climate change as we know it does not happen. The idea of the Anthropocene, and all that it means for how humans understand their agency, is an indirect accomplishment of computational simulations of planetary systems over time.
In turn, the relay from the idea of the Anthropocene to climate politics is based too on a geopolitics of simulation. The implications of simulations of the year 2050 are dire and so climate politics seeks to mobilize a planetary politics in reflexive response to those predicted implications, as i mentioned. That politics is recursive. Deliberate actions taken now consciously to prevent the future. As said, this is an extraordinary agency to give simulations. I doubt that Greta, or other climate activists may like the idea, but climate politics is one of the important ways in which massive computational simulations are driving how human societies understand and organize themselves. It’s why the activists are in the streets to begin with.
Part 1: Intelligence as Simulation / Simulation as Intelligence
That simulations relate so closely to descriptive and predictive knowledge of the world is not surprising given that cognition depends on descriptive and predictive mental models. Simulation is one of the most important ways that human brains work. Our friends from Neuroscience and Artificial Intelligence may raise the point that simulation is how minds have intelligence at all. The cortical columns of animal brains are constantly predicting what will be perceived next, running through little simulations of the world and the immediate future, resolving them with new inputs and even competing with each other to organize perception and action.
Different versions of this general understanding largely converge on the point. Karl Friston’s Free Energy Principle depends on the reconciliation of anticipated and actual perception, Jeff Hawkins refers the recursive feedback between mental models, new perceived novelties update the background simulation used to navigate environments, Computational neuroscience would agree that the basic process occurs not only at the relatively abstract level of cognition but even at the level of individual neurons. Andy Clarke and others explicitly invoke simulation as the modeling process and what makes the noisy flux of perception into a coherent, predictable phenomenological reality. Douglas Hofstater said “yes you are a loop” meaning this model-test-feeback cycle is not just something that intelligence does, it is how intelligence works.
The same goes for collective intelligence as well. Humans’ gigantic prefrontal cortex allows for/ and is the evolutionary outcome of linguistic and strategic cooperation. This allows us to share and communicate counterfactual, speculative, hypothetical scenarios and to coordinate actions to realize or prevent them. Everything from hunting in groups to building spaceships depends on projective simulation, at the individual and collective level.
At the end of the day –literally– we drift away from that coherent perceived reality into the deeply personal world of sleep. Arguably, dreams are another essential kind of simulation necessary for all animals. For humans, one hour of unconscious processing of reality for every two hours of waking life is necessary. Every night you lay prone, frozen, unconscious, lost in a virtual world where mental simulations dance with themselves. Perfectly normal.
Given that simulation is so essential to thought and to individual and collective intelligence, and that this intelligence manifests in the technological artificialization of the world, it is perhaps to be expected that we would make artificialized intelligence based on this very same capacity.
AI is simulated intelligence, which in the form of LLM, uses “language” not just in the conventional sense of chatbots but also as the engine of difference underlying robotics, image processing and genetic sequencing for example.
However, it is worth noting that the psychological and technological development of AI has hinged on a kind of simulation anxiety. For the Turing Test, the standard for intelligence was based on the differentiation of human intelligence from computationally simulated human intelligence. To know if it is a simulation or not would come to one of the ways that the non-simulated (ostensibly us) is defined by this difference. Any uncertainty and confusion that muddies the split makes us less clear who we are.
This is perhaps a psychological version of “model collapse,” the process by which large AI models trained on the output of other models, begin to fold in themselves spitting out nonsense, not unlike humans undergoing a self-recursive nervous breakdown.
Finally, this itself introduces a weird kind of simulation politics, whereby the goal is to redirect a model that governs the world by feeding it deliberately contrived data, thereby bending its perception of the real. Data poisoning is one way to do this, by making original human culture illegible or garbled, but so is flooding the model with fictionalized data that gives the AI a distorted sense of the world that is to the advantage of the actors.
If you want the big model simulation to think something is real or not real, organize potemkin data that makes it seem so. If you want the simulation to think there is a lot of traffic Walk across a road over and over and over. Show up at a favorite spot everyday if you want the simulation to think it is popular. This is the future of politics as gamified recursive simulation: people contesting the meta-model of society by doing things that they think will make the model correspond with the reality they prefer.
Part 2: Toy Worlds and Digital Twins
AI works not just by simulating us but by simulating the world. AI’s, such as driverless cars, are trained in what are called Toy World simulations where they can explore more freely, bumping into the walls, until they, like us, learn the best ways to perceive, model, and predict the real world.
Toy Worlds are where some AI’s learn to navigate the real world by navigating focused, reductive simulations of its contours. But these are not always closed. For AIs the boundary between a simulated world built of data and the real world perceivable as data is not always clear. For those training AIs to support physical actions in the physical world, this fuzziness can be leveraged. In simulated worlds, time can be speeded up, multiple generations and iterations can spawn in an instant.
These Toy Worlds serve as a bounded domain of constrained information exchange and interaction between otherwise unlike and incompatible things and actions. The sim-to-real passage occurs not only in terms of specific learned expertise, but also through the virtual- physical hybridization of direct inputs and outputs, for example, AI’s interacting with blends of both real and virtual contexts and collaborators at the same time.
The AI’s world is a simulation of ours, but one we can interact with. For the AIs, our world is part of the omnisimulation that it calls simply reality.
We have now moved further away from simulations as providing augmented cognitive access to physical reality, and closer to simulation as constructed, fictional, synthetic experiences. In some respects the same principles of simulation hold -they are built with the same software and by and for the same humans after all –but in other respects their purpose, their relation to the world, to grounded knowledge, and to subjectivity and agency are completely different.
Virtual environments can refer virtual and augmented reality, in which a user is immersed in a digitally constructed synthetic world, and also to physical built environments which are proxies for other real places, where one is a test site for the other. In turn, it is not always so clear to everyone involved that they are even in a simulation. Sometimes some people know, but others don’t, and sometimes no one is really sure. Let’s take them one at a time.
Part 2: Virtual Reality 1: Computational Virtual Environments
I would argue that VR is a particular kind of Toy World more than the inverse. That is, Toy Worlds are not just for AI’s but for people too: VR is a kind of digital twin for experience itself. But If scientific simulation allows access to ground truth through computational abstraction, then does VR free experience from ground truth?
Through VR and video games, simulations have become a mass-consumer content platform, providing immersive experiences for communication, gaming, exercise, and just sitting there zen-like as virtual whales swim overhead, as well as expert domains such as collaborative laboratory science and 4D data visualization.
In theory, this means new kinds of creative expressions and experiences, one can place an audience/user to construct or demonstrate a world can now place an audience in the body of a character, but in doing so, the continuity of immersive space, means they lose the ability to steer attention through cuts, shots, and angles. Does narrative give way a form of storytelling that is more procedural, full of nested tasks and contingencies instead of discrete events?
What then are the limits of the kinds of experience that can be virtualized, especially since experience is not exclusively human but shared by many other forms of intelligent life?
And yet, even as it presents designed realities, VR technologies may offer new insights into the real, informing philosophy of mind, the neuroscience of perception, perhaps also necessitating a refactoring of Art History as the physical relations between image and mind are reconciled.
One question we will ask a prompt is when VR becomes more pervasive, what will we call this reality? what do we call non virtual reality? “Baseline reality” is I think the best candidate, the latter making the virtual the referent against which the real is defined.
How this leads us closer or further from the real is an open question and suggests different relations between illusion and reality, not unlike the difference between Plato’s cave and Wayung Kulit shadowplay. Let me give you a few examples of what I mean.
A project that some students spun out of my old lab at UC San Diego is a company called Nanome which makes now very successful virtual reality software for advanced molecular visualization, modeling, simulation and design. Their customers include all the major drug discovery players as well as many synthetic biology companies. The physical reality of protein binding to molecules is perceptually accessible through 3d models with tactile dimension.
However Sometimes, as we know, the simulation is meant to be an illusion.
A corollary technology to VR is Augmented Reality whereby people wear something like glasses that overlay their world with synthetic perceptions, programmed interactive hallucinations, that blend physical and virtual worlds into one. The shadow and the cave merge.
By contrast, John Carpenter’s They Live presents a kind of reverse of augmented reality. Here the main character finds a batch of sunglasses that instead of adding a layer of ideological augmentation to reality, they instead remove and subtract the parts of generally perceived reality that are illusions. For AR, the subject wears the tool to simulate reality, but here they wear the tool to escape the simulation that is constructed reality: critical Theory as Ray-Bans. In this case, they reveal that the world is actually run by alien lizard people who manage humans like cattle. That both AR and They Live scenarios correlate with symptoms of schizophrenia is a theme I will pick up in a moment.
VR has been represented in cinema in many ways, perhaps the most compelling is in David Cronenberg’s Exiztenz, where the virtual world looks, well, exactly like the real world except that impossible things happen regularly.
In many respects, the cinematic experience itself is already an immersive virtual environment as a century of film theory already suggested already. More recently, the ante has been raised by things like The Sphere in Las Vegas which posit not a first person interior VR but, VR as a medium for crowds, truly a mass ornament as Seigried Krackauer called it. To experience this costs about $500 per ticket for 2 hours, which is expensive, made more so by the fact that one has to endure a U2 concert for the full duration.
The mini-Sphere at your local mall, is no less a virtual environment. Now that cinema has been completely absorbed by Marvel, this metagenre is defined not only by superhero plots but also by how its cinematic form is constructed via layers and layers of photographic and computational imputs, composited together into new reality that is both magical and oppressively familiar.
Part 2: Virtual Reality 1: Computational Virtual Environments
As said, the virtual environment may be physical not digital, and it may also be constructed not as an immersive illusion for its own sake but, closer to the purposes of scientific simulation, as a digital twin of another physical place or reality. A classic example is the Bay Area watershed simulation…
Keeping with the cave theme, consider Chauvet, a site in France discovered by accident quite recently, that is full of incredible neolithic art dated to roughly 32K BCE. By comparison, Lascaux is dated to 16K BCE, so consider that Chauvet is as distant in history from Lascaux as Lascaux is from today. You may know Chauvet from Wernor Herzog’s film Cave of Forgotten Dreams. The art is obviously priceless and fragile, and yet also part of human heritage. To allow visitors a chance to experience the art, a second replica of the original cave was constructed nearby that admits human visitors to admire its wonders. It’s not quite a theme park and not quite not one. In case the trip to France is inconvenient, the simulation has a simulation. You can visit the second cave virtually in a narrated VR tour. Shadows have shadows too.
Physical simulations are sometimes constructed when one location is inaccessible from the other, but when there is existential importance in modeling, testing and understanding how systems live and work in the remote site.
Space exploration has been a driving force in the design of these, where a simulation on Earth is a proxy for a place out there, or a test out there is a proxy for something important down here.
You may be familiar with Biosphere 2, the ill-fated attempt to build a sealed virtual human society meant to simulate a future such society on another astronomic body. As NASA had shown, it's better to optimize a terrestrial simulation before sending humans out of the atmospheric epidermis. This simulation was not run by NASA, however, but rather by a very 1970’s theater collective lead by deeply untrained charismatic leaders who did not account for the inevitable invasion of ants and cockroaches and ultimate virtual famine. Arguably, Biosphere 2 was less a failed simulation of a successful moon colony than a perfect simulation of how certain utopian ideas about closed loop microsocieties lead to implosion.
By the way, Biosphere 2 is now run by ASU. They hosted an art show recently where I was invited to submit a work. Half-seriously I provided the idea of a miniature Biosphere inside of the Biosphere, a simulation of the simulation if you like. They were not amused, but the piece was included.
Part 2: Virtual Reality 1: Computational Virtual Environments
In the examples above, all the participants understood that they were in a simulation. They may not spend a lot of time pondering the implications, but they know on a conscious level that what they perceive is artificial in some way. This is not always the case. One of the most important social and epistemological dynamics of simulation is what I call ontological asymmetry: when there is one fraction of participants who knows it's a simulation and another fraction who does not. Often the purpose of the simulation requires this asymmetr.. Somebody has to think this is real, for the virtual experience of the other to go as planned.
The Orson Scott Card novel Ender’s Game is a classic example of this. The young space cadet thinks they are playing a battle simulator at which they have become impressively expert, only to learn that it was no sim after all. He had just blown up whole planets full of people, not knowing what he was doing. The theory was that if he had known it was not a sim, he would not have acted as desired.
Perhaps the goal of fooling the protagonist of the simulation is not to coerce them to do something specific, but to put them in a frame of mind where their desires can wander naturally. Consider Butters. His mates on South Park convince him that they have let him use their new VR headset which includes a perfect sim of their school. They have instead only installed upon him a snorkeling mask, which explains why what he calls “the graphics” are so realistic. This is an example of what we might call a placebo simulation. At least somebody thinks its a sim, but it’s not.
One of cinema’s most famous examples of ontological asymmetry is Truman’s Show, where the main character’s entire life from birth to adulthood is a massively artificialized physical environment, with artificial friends and family who are in on the illusion. Slowly but surely he begins to suspect something is not quite right, exhibits classic symptoms of a particular kind of schizophrenic episode, today called Targeted Individual Syndrome, which ultimately leads the protagonist toward breaking the asymmetry and taking some control of the process of designing his life.
By this measure, the inverse of Truman’s Show is Westworld. Here the protagonists do know it's a sim but all the background characters do not. The asymmetry is reversed. Eventually they do come to realize that the Toy World simulation is actually not a sim after all. Given that these characters are androids, this uneven realization of the real leads to less cut and dry existential lessons than TrumanShow.
In both cases, the issue of “consent” is obviously relevant for such asymmetries. The Milgram experiments in acquiescence to authority are also famous simulations where ontological asymmetry caused psychological harm to those to didn’t know it was a sim. They thought they had harmed someone, when they had not.
Continuing deeper into the abnormal psychology of simulations, there is a strong correlation between simulation and anxiety, that takes different forms. I will explore this more in a moment in relation to personal simulations, but first it also relates to the question of knowing/not knowing and deliberate suspension of disbelief, especially as therapy.
Among the primary uses of VR in the military, besides training for battle and pre-enacting war, is for PTSD, and indeed re-enacting the traumatic experience as a way of exorcising its torment. Reenactment of trauma is basic not only to Freudian psychoanalysis but to mourning rituals the world over.
By contrast, In the TV show, The Rehearsal, anticipated trauma is dealt with by constructing full scale simulations of difficult conversations and encounters and having simulation protagonists pre-enact what it will be like, preemptively repeating the trauma before it happens, so that it doesn’t happen at all.
At the end of this spectrum are the virtual worlds of Wandavision where the scarlet witch deals with the death of her family by using her magic powers to construct fully immersive environments, both physical and virtual at the same time, where she and they can live out their days in various sit-com scenarios. Here it's less than simulation allows for the trauma to be repeated rather than reversed. Reality is an egg unscrambled by the preferred artificial reality.
Part 3: Political Simulationism
Shifting gears a bit, momentarily out of the dynamics of consensual and non-consensual illusion, the theory of simulations would also apply to questions of politics and governance where the distance from consensual and non-consenstual illusion is not always so vast.
Simulations play a big role in how States and other governing institutions imagine a role for everyone and everything. They give a sense of control through visual coherency and enclosure. It seems as though the complexity of the phenomena simulated is accounted for because the elegance of its interfacial reduction is convincing. Often this works just fine. You don’t or want scientific precision in modeling social systems. It also depends on what kind of simulation it is: descriptive, predictive, or projective. In many cases, what is needed most is the representation of coherency as institutional rhetoric.
Speaking of which, one of the essential techniques of organizational futurism, especially during the Cold War era was and remains scenario planning: a form of official, institutionalized reflexive simulation of potential political and military realities. Back in the day, as wielded by Rand Corp, Herman Kahn, Pierre Wack and others, scenarios were one of the key ways that governments, corporations, and militaries would model not the determinant future but the contingent space of possible futures, rendered for executive consideration in the form of just-so science fiction stories often told with non-fiction rhetorics.
They could be called recursive in the sense that the indeterminancy was based on the understanding that following the simulation to its logical conclusion itself causes a particular future to happen, rather than merely predicts it. In this sense, they are proactive and normative, implicitly recommending the manifestation or prevention of a given scenario.
This delicate contingency provoked the curiosity of people like John Cage to link their esoteric interest in chance with the composition of artworks based on similar scenario design techniques. Via people like Wack, the overlap between the technocrats and the artists and mystics was quite strong.
The relation between politics and simulation is not merely one of futurism; it is also with the emergence of planetary computation, a realtime medium for the maintenance and gamesmanship of adversarial relations. As the world has grown increasingly interconnected, effective governance has become contingent on complex simulations of global systems.
These simulations serve not only as a tool for human sense-making, but as algorithms for creating policy, distributing resources, and managing trade-offs. Simultaneously, the deployment of earth observation satellites has enabled the collection of diverse, continuous data streams to power these simulations. While remote sensing has made it possible to comprehend the Earth as a planetary system, it has also revealed the potential to leverage the same simulations for economic and geopolitical influence.
In the competition for accurate simulations, various counter-simulation techniques arise to distort and degrade the simulations of other actors. Behavior is hidden behind camouflage, senses are jammed, decoys distract attention, while misinformation proliferate. While the inflections can be vastly different among various actorst, the stakes are ultimately about governance and sovereignty: who is able to maintain accurate pictures of reality, who can act on those pictures, and who can distort the simulations of other actors to their own advantage.
Part 3: Simulation Before and After Violence
There is no way to properly summarize the politics of simulation without speaking its role in political violence. Allow me to introduce Fort Irwin, in the desert East of Los Angeles, a simulated “Middle Eastern” city which the US military trains for, among other things, simulated communication with confusing local merchants, and ultimately full-scale urban combat, complete with extremely loud and real explosions.
Other simulations are constructed, not in anticipation of violence, but after the fact. They seek to recreate, reconstruct, or reenact the violence, and do so for intricately different purposes.
Forensic Architecure, a group associated with Israeli, now British, architect, Eyal Weizman, deploys all the tools and tricks of architectural modeling and visualization to the reconstructive simulation of purported crimes of the state. The practice tries to strike a balances between art and law. Sometimes the same work is submitted into the public record as part of a prosecution against the accused, hence the name forensic, and sometimes it is shown in prestigious galleries and museums. The ending of each of their case studies is always the same: the bad guys are guilty. This underscores the fluidity by which simulations slide between domains: not only are facts posed as fictions, but fictions are a way of deciding the real.
These two forms both use narrative and projection in different ways. Scenario planning is a narrative prediction that becomes normative by its implementation. Forensic architecture, by comparison, is a descriptive simulation of causal processes that becomes normative through narrative and rhetorical advocacy.
However, the relation between simulation and political violence is not always so simple.
Consider two of the most extraordinary films of the past century, Gillo Pontocorvo’s Battle of Algiers 1966 and heimer, The Act of Killing 2012. Another whole lecture is due here but suffice, that Battle of Algiers was made just a few years after Algerian independence and recounts the rise of the FLN as a revolutionary force and the French response. The film starred Saadi Yacef, one of the actual FLN leaders, playing himself. Street battle scenes were recreated and shot newsreel style often in the same location in which they had taken place. American releases of the film began with a disclaimer, insisting that “not one foot” of newsreel or documentary footage was used. The film-as-model worked in two temporal directions. Battle of Algiers simulated the violence of the previous years but also served as a model for future revolutionary action, and was even screened by the Pentagon after 9/11 as a way to understand insurgency in the Islamic world. Whereas Forensic Architecure’s simulations are geared toward preventing similar events, Battle of Algiers as more of a hyperstitional template – though history has a way of reversing these..
We began in Indonesia and now return there.
The Act of Killing also reconstructs postcolonial revolutionary violence from the Cold War era, but this time from the Right. It is a surreal, chilling retelling of the mass killings in Indonesia of upwards of a million suspected communists that took place during the same period in which Battle of Algiers was shot and released. Instead of simulating the events ex post facto as way to celebrate the bravery of the fighters or to identify the guilty parties, the film casts the now aged perpetrators in the role of their younger selves (a al Saadi Yacef) within elaborately theatrical recreations of their murderous acts. Here the simulation is closer to the repetition of trauma, but in this case that trauma is itself asymmetrical. For many of the perpetrators who are all too humanly oblivious to the significance of their acts, and victims and their families who are tormented by memories, the simulation works, sometimes, to reconcile the meaning the violence.
Part 3: Back to the Shadow: Persona
We move now from the past to present, from the filmic simulation to the computational and from collective historical simulation to the individual and personal.
The formations of digital identity under planetary-scale computation are aligned toward the individual human person as the base unit of sensing, analysis and recursion. We confront the digital shadows that each of us produce and are produced by, and which we both possess and are possessed by. These shadows emerge through processes of doubling, tripling, reflection, opacity, transparency, individuation, massification, desire, control, erasing, staging, framing, deframing all at once. We have no choice really but to approach them with trepidation, fascination, curiosity, dismay and satire all at once.
The politics of simulation can also be very personal. As you pass through a security gateway— perhaps at an airport— what is under inspection is not only your physical person, but also trace digital personas linked to you but which live in a near-distant shadow city called the Cloud. If the man in the uniform lets you pass, it’s because a decision was made according to risk models on those silhouettes of which your physical person is a reflection. Your ears may burn as the infrastructure whispers about your doubles, but it’s not just you in play.
The profile as shadow is a product bought and sold. Consider that The idea of celebrity is already a kind of simulated person, an artificialized persona that can split from the original person and take on a life of its own. This can cause psychological stress, and this can be monetized.
In 1971, Stanislaw Lem’s Futurological Congress imagined a future in which hollywood films would be populated solely with licensed synthetic doubles of actors. In 1974, Rem Koolhaas and Rene Daalder, wrote an unfilmed screenplay called Hollywood Tower that was based on the same premise. In 1975, Lou Reed was touring Australia and became very annoyed with an interviewer and disclosed the following…
More recently, moore’s law has allied the prophecy of full celebrity simulation to become more advanced. Consider the geriatric rock band KISS who have officially now retired but recently announced that their digital avatars will be going on tour in their stead. Or the French perennial presidential candidate Jean-Luc Melanchon, who mildly impressed the crowd by appearing in person in Lyon while as a Tupac-style hologram in Paris at the same time. Obviously the actors and writers strike put Hollywood on pause while an entire industry organized around the mass production of digital persona, and wrestled with the macroeconomic shift in labor power from those who create tranining data to those who create models.
There are ghost cities populated by legions of mimetic personas crackling within nearby data centers even as the surrounding landscape is largely unpopulated by physical human beings. Any such may be home to hundreds of millions of shadows but only a few dozen workers. This ratio is a sign of things to come. It is a posthuman urbanism in practice but not in theory (even shadow cities came to be an predominant urban form, their progress largely went ignored in architectural schools in the first decades of the century when increasingly fantastic accommodations for human clients took precedence.) This does not mean that shadow cities are actually virtual. To the contrary, the sprawling distribution of factories, ports, container sorting centers, freight airports, as well as the networks of thirsty data centers comprises a discontinuous megacity for objects. To imagine it as numinous is an expensive illusion.
There is “a there there,” but this “there” is also right here. When you and I chat and post to one another, I am in one place and you another. We may even be in the same city at the same time, but the conversational point of contact between my persona and yours is, literally and physically, located in a shadow city where neither of us live. To converse, we draw upon shadows and speak to one another through them as masks, carving links between human zones and shadow cities. We sew threads between one another and between places, and in doing so contribute more texture to the model simulations that mediate these circuits: sprawling, interlocking and incommensurate.
Conclusion: Simulation Anxiety
This talk is as mentioned and invitation toward a more general theory and so will conclude not be solving the puzzle but by making it a bit more complicated. I will conclude with one last theme: simulation anxiety.
Simulation anxiety means a different things. First it is anxiety about whether or not something is or is not a simulation. Philip K Dick characters are archetypes for this, but their reality-questioning angst is not the only form this can take. Ultimately, in societies where simulations are foundational for political, economic, logistical, ideological and identarian formation, anxiety as such can take the form of simulation anxiety.
Instead of simulation healing trauma through staged repetition, the uncertain relationship between the stage and staged induces, if not trauma then at least anxiousness. This then may in turn need to be dealt with by future simulations, about which there is some uncertainty, and so on.
For example, the Art world’s financialization of affect has long been concerned with identification of original and forgery, so much so that the forgery as satire became an established format in and of itself, from Duchamp, to Warhol, to Richard Prince, to Thomas Kinkaide. Sometime its collectors are worried about investment, and sometimes its consumers are worried about authenticity. Consider the streets of Santa Fe, NM where native american artisans, mostly Navajo, sell beautiful jewelry, some of which is made in a factory used to mass produce hand crafted objects. Today, however, that factory has Navajo owners. Is this authentic or not? Who’s to say, but what value does the object have if it does not embody unadulterated authenticity? For whom?
What about the authenticity of simulated humans? In the 1920’s, Makato Nishimura was so distressed by the human simulations he had seen in Capek’s play RUR, that he designed Gakutensoku, a humanoid robot made without metals that represented not the degraded, machinic qualities of the body, but it’s higher spiritual capacities. It could open and close its eyes, move its head, at least before it was lost for good while touring Germany in the 1930’s.
Compare this anxiety over the missimulation of the human with that of Turing’s Test, where the very status of the human would come to be defined by all the ways, ever shrinking, that is it is different from, not similar to, its artificialized other. For Gotutensaku, anxiety seeks resolution and union between the human and its shadow and but for Turing Test, to keep them forever bordered.
From rubber sex dolls accosted by the lonely to Paro the robotic seal given to elderly dementia patients in Japan, our anxiety is here not with whether or not we can tell, but whether or not those using them really know or really care enough if it is a simulation or not. Does the dementia patient know it’s not a real seal that seems to love them? Do they know and not care? Do they care and not know?
We we interact more and more with traces of our simulations of self, our digital shadows and our replica personas, does this make simulation anxiety more acute? The more that we are asked to verify our originality, is the more our existential status put into question?
As for the simulation politics of gaming the model by performing what you want it to think, this Hawthorne Effect ontology become so normalized that it expands the potential space of simulation to the edge of reality itself, engendering a new kind of deja vu effect, called ‘glitch in the matrix’ where some kind of anomalous regularity suggests that the big operating system in the sky may have copy/pasted the NPC’s too much and, like Truman’s revelation, given the game away.
It’s hard to keep it all straight. As for simulation anxiety episodes like the Moon Landing, seen here as a simulation of a simulation in the film Capricorn One, in which the Moon is played by Mars, and Neil Armstrong by OJ Simpson, we should remember that as the most watched television broadcast in history up to that point, the moon landing was shown by the largest American network, CBS, as a simulation. It said so right on the screen, but as all those people’s memories of the landing, and of the broadcast, started to fade over the years, a kind of inside out Mandela effect may have taken hold, people half remembering the simulation they really did see as a simulation they didn’t see.
I hope that I have provided some good training data.
I will offer just a few closing thoughts.
If our era is in fact one defined by simulations, one piece of evidence for this may be the fact that we use simulations for so many different important things, and yet simulations as such are not widely recognized, acknowledged or discussed, but rather very much taken for granted. The fish does not bother to theorize water.
If we were to take them more seriously, however, and be more reflexive about how we used reflexive simulations, for example, this would, as I hope is clear, mean that we would be thinking about how we are thinking, which may, be the basis of some deliberate reorientation of our own purpose.
This would touch on many things but would have to include, as I, among many other things, the different between reflexive and recursive simulations, the dramas of ontological asymmetry, placebo simulations, the divergence of scientifically deduced reality and computationally generated illusions, the relation between Model and intelligence, modeling as intelligence, intelligence as predictive, modeling, and, of course, the additive microtraumas of simulation anxiety.
Lastly, One of the key ideas of the Antikythera program on the future of planetary computation is that at some points in history, our ideas are ahead of what is technologically possible. At these moments, utopians and various avant-gardes make the plans. At other times, technology outpaced the ideas. Technical capability is way ahead of theory. This is where we are today. As such the work of philosophy changes. It's less about projecting critical truisms onto technology than a direct encounter with, in this case, computation and computational simulations, that would produce the new concepts needed to orient not only the model but ourselves. This is why simulation –a critical practice without a sufficient theory– needs our attention.
Thank you.