Søren Pold

Multimedia Studies and Comparative Literature
University of Aarhus
pold@multimedia.au.dk

 

Until now, digital arts have largely been understood to belong in traditional genres or forms of art: we are said to have electronic literature, net.art, or electronic, techno music. Sometimes interesting discussions have arisen concerning the very ontology of digital art, and questions such as whether net.art should be seen within the tradition of the visual arts or as a form of literature connecting it to the different traditions and questions regulating these arts.¹While it is not my intent to disconnect digital art from traditional art, or to argue for the irrelevance of the traditional forms of art, I will in the following establish the interface as an aesthetic and critical framework for digital art.

 

The interface is the basic aesthetic form of digital art. Just as literature has predominantly taken place in and around books, and painting has explored the canvas, the interface is now a central aesthetic form conveying digital information of all kinds. This circumstance is simultaneously trivial, provocative, and far-reaching–trivial because the production, reproduction, distribution and reception of digital art increasingly take place at an interface;² provocative because it means that we should start seeing the interface as an aesthetic form in itself that offers a new way to understand digital art in its various guises, rather than as a functional tool for making art (and doing other things); and, finally, far-reaching in providing us with the possibility of discussing contemporary reality and culture as an interface culture.

 

In what follows I pursue these three lines of thought in order to outline the interface as an aesthetic form. I start by taking a brief look at the explorations of the interface undertaken in the fields of engineering and computer science in order to sketch out the traditions that dominate the design, functionality, and cultural conceptions of the interface. In these traditions, realism is the keyword–not a realism found in aesthetic tradition, but a realism that stems from the pragmatic urge in engineering to deal with the physical world. I then confront this realism with aesthetic realism and the question of how digital artworks can be seen as a reflection on and reaction to this. Finally, I analyze a computer game (Max Payne), a net.art game modification (Jodi’s SOD), and a software artwork (Auto-Illustrator) to show how they engage with the interface and what they make us see through and in it.

 

The Engineered Image

 

The graphical user interface (GUI) as we know it does not stem from an aesthetic tradition, but from an engineering tradition that has paradoxically tried to get rid of it. Until recently it has largely been understood in technical terms and developed in engineering laboratories. Important starting points were Ivan Sutherland’s first graphical, interactive interface in Sketchpad (1963), Douglas Engelbart’s Online System NLS (1968), which introduced information windows and direct manipulation via a mouse, and Alan Kay’s development of the desktop metaphor and overlapping windows.³

 

Human-Computer Interaction (HCI) research continues this engineering tradition. From the beginning, HCI aimed to increase the “user-friendliness” and “transparency” of the interface and over the years has involved cognitive sciences, psychology, ethnographic fieldwork, participatory design, etc. A leading usability expert, Don Norman, explains the desire to eliminate the interface in “Why Interfaces Don’t Work”:

 

The real problem with the interface is that it is an interface. Interfaces get in the way. I don’t want to focus my energies on an interface. I want to focus on the job. . . . An interface is an obstacle: it stands between a person and the system being used. . . . If I were to have my way, we would not see computer interfaces. In fact, we would not see computers: both the interface and the computer would be invisible, subservient to the task the person was attempting to accomplish. (209, 219)⁴

 

Making the interface, its expression, and materiality more functional and transparent has been key to interface design and the accompanying academic discipline, HCI. In the broader cultural and social understanding of the computer, the tendency has been to understand the interface as transparent, preferably invisible, in order to produce a mimetic model of the task one is working on.⁵ Interfaces should be intuitive and user friendly, should not “get in the way” or otherwise be evident or disturbing. This has led to development of the ideals of direct manipulation and the WYSIWYG (What You See Is What You Get) slogan for the GUI, which became a leading sales argument for the Apple Macintosh from the mid-1980s.⁶ Perhaps the apotheosis of the transparent, invisible interface was Virtual Reality (VR), which was widely believed to be the next interface paradigm from the mid-1980s to about the mid-1990s, when it gradually lost steam. In the dominant perceptions of VR, the interface should be all encompassing and three-dimensional, and the user should be surrounded by an immersive, total simulation. The interface would thus simultaneously disappear and become totalized. Currently, some researchers in the field believe that the future of the interface lies in pervasive computing and augmented reality, envisioned as a new version of the all-encompassing interface, while others have more modest expectations.

 

The Realisms of the Interface

 

If the computer and the interface really had become truly invisible and transparent, computers would mingle almost seamlessly with the world as we know it–perhaps making it a bit “smarter.” If this were true, digital technologies would probably not have any paradigmatic effect on culture and aesthetics since they would not make a marked difference, but of course reality has proven otherwise, and we can now begin to acknowledge the massive cultural and aesthetic impact of digital technologies. It is my view that we can acknowledge this impact through digital art: how the interface changes what and how we see, how we experience and interact with reality, and how this reality is reconfigured through the computer. Digital art in general shows us the role of the interface and the significance of the interface as an aesthetic, cultural, and ideological object. Through reflexive and self-reflexive moments and strategies, digital art foregrounds the interface in ways that traditional software normally does not. Consequently, digital art becomes an important witness to the changes the computer has brought and is still bringing to our societies. Digital art and aesthetics can also help inform the HCI research field with regards to cultural and aesthetic aspects of the GUI.⁷

 

In Interface Culture (1997), Steven Johnson argues that the interface is the most important cultural form of our time, comparing it to postmodern cultural and aesthetic forms. Johnson argues that we live in an interface culture, and that the shift from analogue to digital is “as much cultural and imaginative as it is technological and economic” (40). Lev Manovich develops this idea with his notion of the cultural interface, and artists, conferences, journals, and designers have started taking up the interface and its cultural, aesthetic impact.⁸

 

What is an interface? The purpose of the interface is to represent the data, the dataflow, and data structures of the computer to the human senses, while simultaneously setting up a frame for human input and interaction and translating this input back into the machine. Interfaces have many different manifestations and the interface is generally a dynamic form, a dynamic representation of the changing states of the data or software and of the user’s interaction. Consequently, the interface is not a static, material object. Still it is materialized, visualized, and has the effect of a (dynamic) representational form. In this way, digital art corrects the ideal of transparency. Instead of focusing only on functionality and effects, digital art explores the materiality and cultural effects of the interface’s representationality. What are the representational languages of the interface, how does it work as text, image, sound, space and so forth, and what are the cultural effects, for instance, of the way it reconfigures the visual, textual or auditory? How does the interface reconfigure aesthetics and what does it do to representation, communication, and, in continuation of this, the social and the political? These are all important questions regarding the aesthetics of the interface, but one should not leave out the engineering roots of the interface, both in order to understand the full context and consequences of interface art and in order to see how digital art can feed back into its development. Therefore, postmodern and poststructuralist aesthetic theories should be balanced with the pragmatic realism of the engineers.

 

Interfaces are often implemented as window screens with keyboards and mice, the so-called WIMP devices (Window Icons Menus and Pointers), but other information displays and interaction devices are of course possible, from different interface designs to different input and output devices such as VR helmets and gloves, the ambient displays and devices imagined within the domains of hybrid architecture, ubiquitous computing, augmented reality, pervasive computing, and so on. The interface aims to give us insight into the machine and its dataflow and can thus be understood as a realistic representation along the lines of the engineering tradition outlined above. It is most likely the predominant form of realism in contemporary culture, both conceptually and stylistically; however, it is far from naïve realism. As Johnson writes,

 

put simply, the importance of interface design revolves around this apparent paradox: we live in a society that is increasingly shaped by events in cyberspace, and yet cyberspace remains, for all practical purposes, invisible, outside our perceptual grasp. Our only access to this parallel universe of zeros and ones runs through the conduit of the computer interface, which means that the most dynamic and innovative region of the modern world reveals itself to us only through the anonymous middlemen of interface design. (19)

 

This suggests a different understanding of the interface’s realism than is suggested by the transparency ideal and WYSIWYG. The interface is, as suggested by the quotation from Johnson, rooted in an active and dialectical relationship between reality and representation, the interface entering in front of–or perhaps more correctly instead of–an increasingly invisible reality.⁹ Here we are closer to aesthetic realism–for instance, literary realism, which is not based on a naïve mimetic faith in the possibility of reproducing or depicting reality, but rather on a loss of immediate reality with the rise of urbanity, capitalism, and modern media. Consequently, aesthetic realism in broad terms represents an active and constructive reaction to a heterogeneous, mediated, complex, and symbolic reality.¹⁰

 

The interface aims to visualise invisible data. It is a new kind of image originating in an engineering tradition and can be understood as an extension of instruments like radar and scientific tools, which do not represent any analogue image of reality but rather sheer data.¹¹ As formulated by Scott deLahunta, the interface is “more information than likeness; more measurement than representation.” Consequently, realism is the dominant representational mode of the interface, even though it is a complex, informational, and postmodern realism.¹² In the following, I shall point out three rather different kinds of interface realism: illusionistic, media, and functional realism. They are not mutually exclusive but are often coextensive and combined–for example, as layers in the same interface. The realist agenda is set by the engineering tradition; I will discuss how the three artworks reflect this realism from three different perspectives. Consequently, I shall argue that these artworks offer a realist perspective, even though at least some of them might not look traditionally realistic at first glance.

 

Illusionistic Realism: First-Person Shooter

 

Illusionistic realism drives the computer game industry and the demand for improved resolution and graphics, more realistic scenery, and better sound cards. In addition, the interfaces of computer games have developed toward a real-time generated 3D space, which is often seen from the perspective of the player and/or his or her avatar in the game. Since Wolfenstein 3D (1992) and Doom (1993), the First-Person Shooter (FPS) genre has developed into the Third-Person Shooter sub-genre, in which the avatar becomes a character with a personal story and a fictional perspective.

 

The FPS perspective is in certain respects the apotheosis of the illusionistic interface. It derives from simulations and VR in its enactment of “the subjective, point-of-view aesthetics that our culture has come to associate with new media in general” (Bolter and Grusin 77). As Bolter and Grusin say, the representation strives toward immediacy.¹³ The FPS not only inherits the visual perspective, but also the ideologies that come with VR, which can be described as a super-individualized and apparently omnipotent linear perspective–a visualization of the bourgeois free individual.

 

Traditionally, the FPS perspective is determined by the shooting range of impressive, phallic guns. The player perceives and interacts through oversized guns whose gendered implications are obvious. Indeed, the FPS genre can be said to overstate and in some cases even invert the ideal of the free bourgeois individual; one striking example of this is the gridlocked noir detective of the computer game Max Payne (2001)–a game marketed as “realism to the max.”¹⁴ However, the illusionism of the game is gradually reversed to offer an experience of mediated perception.

 

The game’s narration starts at the very ending, and the player has to struggle for hours to reconstruct the plot in order to return successfully to this ending. As is typical of the genre, the main character and player-avatar, Max Payne, is caught in a stratified maze controlled by drug lords and corrupt police, and only a combination of clever tactics and desperate shooting will get him out, after which he is trapped in the next level. The player is caught in a space and a plot that are clearly controlled by others–by the mobsters and the urban spaces on the level of the plot and by the cybernetic game engine on the structural level.¹⁵ In this sense the cyber-noir narrative of Max Payne is a reversal of the ideals of interactivity as a way of setting the user free and giving him or her control.

 

A closer analysis of Max Payne supports this view, which I shall elaborate in order to show how the experience of the interface is also a narrative experience. Max Payne seems almost like a comment to the heated discussion around the functioning of narrative in computer games.¹⁶ As mentioned, the game starts at the ending, and the player has to move back in story time in order to (re)construct the plot.¹⁷ The plot follows classical Aristotelian or Hollywood standards with three acts–a beginning, middle, and end–a point of no return, and increasing narrative tension. The story–and the game play–appears at first to be about exploring, enacting, and reconstructing a plot that is static, in the past tense, linear, and prescripted. Already through this immediate first level of the narrative it becomes clear that the game does not aim to fulfill the notions of interactivity. In any case, the player does not interactively control the plot; rather the game only allows the user to investigate and carry out an already firmly established plot. Just as the game inverts the ideologies of mastery and the sovereign individual inherent in the first-person perspective, it also runs counter to the libertarian notion that interactivity empowers the user. Rather than controlling the story, the player is trapped by the plot.

 

The narrative serves more or less as a frame or motivation for the game play, but this is not all it does. Already in the third chapter of part one, a second meta-narrative layer is introduced. We know the ending and we are also quickly introduced to the usual clichés of Hollywood narration: the sudden destruction of the family idyll, the strong narrative conflict–even the weather is horrible in order to emphasize the sinister setting. In this sense we are less primarily engaged in the narrative; instead, the game designates the narrative as a cliché. All the signs of Hollywood narrative in an extreme, clichéd,noir version get thicker and thicker until they more or less get in the way, in the sense that they point toward narrative structure in general rather than support a particular narrative. In chapter three we are told that we are now “past the point of no return,” and the construction of the narrative is consequently laid out so explicitly that the game could easily be interpreted as a self-conscious intervention in the ongoing debate about the roles of narrative in computer games. Narrative becomes an effect to which the game alludes self-consciously and which it puts on but does not fulfill in the deep Aristotelian way imagined by the proponents of interactive narrative, such as Janet Murray. Instead narrative is only a surface or skin; it does not attempt to become hegemonic or to account for and relate to all aspects of the game, but, like postmodern novels and cinema, it alludes to narrative, quotes it, and deconstructs it, without fully enacting it. If narrative is a grand master-scheme that secures coherence, Max Payne at once enacts and challenges it.

 

Hence, the game engages a meta-narrative that is explicitly conscious of its noir genre, even pointing out intertextual references: chapter three is called “Playing it Bogart” and other later references include “Noir York City.” Max Payne describes himself as someone who “had taken on the role of the mythic detective, Bogart as Marlowe, or as Sam Spade going after the Maltese Falcon.” Indeed, the main character, Max Payne, slowly reveals himself as just a narrational character–a role or a function in the narrative–which is highlighted by the symbolism of his name. Already in the beginning we are told to go back “to the night the pain started,” and in chapter three Max Payne enters a hotel “playing it Bogart,” while the “cheap mobster punks” there make fun of his name (“it’s the pain in the butt” and “pain to the max!”). It is no coincidence that Max Payne inherits a noir style, which from its outset was a media hybrid. Already in the noir cinema and novels of Raymond Chandler, noir was clearly a media hybrid of filmed novels and cinematic literature often set in a thoroughly cinematographic Los Angeles.¹⁸

 

Through the overt character of the noir detective Max Payne we enter into what can be seen as the third level of the narrative, which is the mediated story or the story of mediated perception and recognition. Here we are closer to the story level, the game play, and how it mediates between the plot and the telling of the story. First we recognize the role that media such as newspapers, radio, and television play in creating a bridge between the plot and the story. Through various news clips we discover that the police are chasing Max Payne. Second, it is obvious that Max Payne is often hindered in seeing and sensing by darkness, fire, dream sequences, or drugs, which point out the mediated perception of the FPS by exaggerating the normal navigational difficulties.

 

In this story of mediated perception, Max Payne is our senses, and he can endure a maximum amount of pain (if we just remember to grab some pain killers on the way), though his pain is only mediated by a red bar in the interface and narrated by his voice and through his frequent deaths–the ultimate failure of the story, after which the player has to reload a saved game and start the scene again. This mediation of Payne’s pain demonstrates the very perspective of the FPS game, at once mediated and illusionistic. The beginning of the third act, where Max Payne is drugged, shows this particularly, since the perspective is strangely widened and the movements slow down–defamiliarizing effects which emphasize and therefore demonstrate the machinery of the normal game perspective or interface.¹⁹

 

The interface thus supports and becomes an important part of the story of mediated perception. The apparent immediacy of the illusionistic FPS interface is gradually but effectively turned into a hypermediated experience through the difficulties of navigating and seeing in the space of the game. This hypermediated experience is addressed explicitly in the first chapter of part three. In a graphic novel sequence in the drugged beginning of the third act, Max Payne first realizes that he is a character in a graphic novel and, in a strange, defamiliarizing repetition, that he is nothing but an avatar in a computer game: “Weapon statistics hanging in the air, glimpsed out of the corner of my eye. Endless repetition of the act of shooting, time slowing down to show off my moves. The paranoid feel of someone controlling my every step. I was in a computer game.”

 

Figure 1: Max Payne Deconstructing the Illusionism Screenshot taken from the video game Max Payne. Max Payne is a trademark of Rockstar Games, Inc. ©2001 Rockstar Games, Inc. All rights reserved.

 

Like the noir genre in general, Max Payne performs, as the Los Angeles-based cultural historian Norman M. Klein says, a “transformational grammar,” turning the utopian notions it enacts into their opposites (80). Consequently, we should categorize Max Payne as illusionistic media realism–realism that simultaneously engages in illusion and is a self-reflexive exploration of its own representational techniques and media. Max Payne is more than illusion. One gradually sees the media, techniques, aesthetics, and genres of the game in a process whereby it, though still engaging in illusion, gradually reveals itself as a self-reflexive and genre-conscious art form.

 

Media Realism–Murder in the Museum

 

The multimedia computer was introduced at the beginning of the 1990s as the result of an almost iconophilic belief in the power of “natural,” “universal,” and “intuitive” images. The advent of net.art can be regarded as a reaction to the iconophilia of the computer industry–as some sort of digital iconoclasm. Jodi, one of the pioneering groups of net.artists (recently the first net.artist to be canonized by a retrospective gallery exhibition[20]) is constantly aiming at demystifying the images on the screen through their works, showing the codes behind the screen and revealing the normally hidden flow of codes that the user interaction causes. In what often look like beautiful crashes, they show the symbolic and coded flipside to the interface.[21]

 

Figure 2: Jodi’s Sod Image used with permission of the artist.

 

Since the year 2000, Jodi has made a series of game modifications. If the mediated and hypermediated quality of Max Payne comes into view gradually through its illusionistic interface, Jodi’s FPS game modifications focus directly on the mediation–so much so that it is sometimes hardly possible to see the game at all. In one of their first games, Sod, the usual dungeons from the FPS classics Wolfenstein and Doom are changed to abstract white rooms, with geometric figures and patterns instead of Nazis and monsters.[22] “Murder in the museum” is what one could call their modification of the classical FPS to the white cube of the museum. The walls have been stripped of their textures, and what turn out to be doors between the various rooms look like one of the major works of abstract art, Malévic’s Black Square. The player is located in a suprematist, black-and-white art museum with a finger on the trigger. It is possible to navigate this abstract space, though it is impossible to see the walls; sometimes they are actually transparent. The basic FPS characteristics are still intact–the player navigates spaces and attacks or is attacked by enemies (appearing as two triangles)–but the experience is rather different from that of more traditional games. Jodi has dropped the illusionistic texture, thereby emphasizing that both space and interaction are based on abstract codes, algorithmic structures, and cybernetic interaction.

 

In the case of Sod, reducing the computer game to geometry and algorithms comments on the illusionism in traditional games. However, it can also be seen as a comment on the debate about the value of computer games, a debate that recalls the conservative resistance to comic strips, rock ‘n’ roll, youth culture, and so forth. Sod is Wolfenstein stripped of its Nazi violence, turned into abstract art and placed in a “safe” context: the art museum. It is the computer game as “Art.” In this way Sod mocks the art world and high art; Jodi’s ironic modification also pays homage to computer games and gaming culture. Jodi do not turn their backs on popular game culture, but rather explore it, investigating its form and tactics; they do not stay in an autonomous art-related medium, but rather turn the aesthetics of the computer into art.

 

Figure 3: Jodi’s A-X Image used with permission of the artist.

 

After Sod, Jodi made Untitled Game, a series of abstract computer games based on Quake, another classic FPS.[23] The series ranges from games taking place in an abstract system of coordinates, flickering moiré interfaces, glitch-like remixes of the original game to games that simply print the code controlling the interface on the screen instead of executing it. All these games can be considered critiques of the naïve illusionism that governs much of the computer graphics and computer games industries. But perhaps what is more interesting is how they demonstrate the elements of the interface, strip off the texture, and show the abstract geometry of digital space–how they demonstrate the materiality and mediality of the interface. The players often lose their orientation, but in turn they are given a unique experience of an abstract, modern, informational, and cybernetic space, an experience which in many ways gets closer to the computer and the functioning of the interface by creating a distance to the illusion. Jodi’s games point toward a sort of computer game that goes beyond the illusionistic, uses the interface as an instrument, and plays with its materiality. The interface, contrary to much HCI wisdom, does get in the way and become an obstacle–in fact, the interface is the experience and what you see is what you get!

 

Media realism in net.art is often dominated by formal investigations; nevertheless, it is still realism in the sense that it deals with perception and sensation at the interface, though from a starting point directly opposed to the illusionistic realism described above. Whereas illusionistic realism starts with our senses and with the way sensation has been addressed in earlier media such as the cinema, and then collides with the forms and potential of the interface, media realism starts with the medium and then tries to show how it collides with our senses, knowledge, cultural forms, and expectations.

 

Functional Realism–Don’t Push this Button

 

Besides the visual dimensions of illusionary realism and media realism, there is another, more active dimension of realism that becomes especially important for computer users.[24] Functional realism in HCI regards the computer and its software as tools, as something “ready to hand.”[25] This functional perspective has produced software we use every day without thinking consciously about it. In general, usability engineering is about designing interfaces so we can use them instead of focusing on them. With functional realism we are perhaps closer to the engineering heart of HCI than with the more cultural and entertainment-related illusionistic realism. Functional realism is about control and functionality rather than illusion and immersion.

 

Frieder Nake, one of the pioneers of computer graphics, perceives the computer as an instrumental medium that we use as a tool while communicating with it as a medium. This constant shifting between the instrumental use and the communicational and representational functions of the medium is inherent to most computer applications. Even computer games and net.art have an instrumental dimension related to navigation, interaction, and functions such as “save,” “open,” or “print.” Functional software tools also have representational dimensions, often as an intrinsic part of the very metaphor(s) behind the software design.

 

It would be easy to dismiss the functional dimension of the computer as irrelevant to art and aesthetics, or to take belief in the power of representations for naïve realism–a belief that aesthetic theory routinely deconstructs. Still, functional realism is part of the computer interface, of its conceptualization, its visual aesthetics, and our understanding of it. Following Nake’s concept of the instrumental medium, we can see the computer as a new kind of machine that mediates the instrumental or functional and functionalizes the representational medium. This dual quality is turned into a standard mode of expression–for example, in digital images on the web and in multimedia that, besides their representational character, are embedded with hidden buttons, objects, and scripts.

 

The functional interface and the functions of software have been investigated in art-related projects, especially in the emerging genre of software art. Digital artists have of course always used software, but in continuation of net.art’s media-realistic investigation into the very medium of the networked computer, digital artists have started working with software as an artistic medium. According to Cramer, software as an art form has some roots in conceptual art and early experimental software development. In 1970 the art critic Jack Burnham curated an exhibition called “Software” that included both conceptual art and experimental software. Cramer claims that this was the first conceptual art show, but it also included a prototype of Ted Nelson’s hypertext system, “Xanadu,” apparently the first public demonstration of hypertext. In both software art and conceptual art, the artist primarily creates the frame for an artwork, the concept. However, in software art, the concept (programming) includes both notation and execution–in other words, both the concept and its potential implementation (see Cramer, “Concepts, Notations, Software, Art”).[26]

 

Besides having roots in conceptual art and literature, software art is a development of tendencies within net.art.[27] Artistic web pages were first executed in ordinary browsers like Netscape and Internet Explorer; around 1997 net.artists started to include in their artistic work the software that showed their pages. Some net.artists started working with web browsers, an early and significant example being I/O/D’s The Web Stalker, which has had several successors.[28] Software art is already established through annual festivals like the Read_Me festival curated by Olga Goriunova and Alexei Shulgin.

 

One of software art’s groundbreaking works is Adrian Ward’s Auto-Illustrator (2000-2003), which investigates how software as an apparently functional tool also serves as a representational structure that influences culture, aesthetics, and art.[29] Auto-Illustrator demonstrates how the instrumental is also mediated, according to Nake’s understanding of the computer as an instrumental medium. It is a complete and fully functional drawing program that uses generative algorithms to draw autonomously as it responds to interaction with the user. In one feature, the user places a number of “bugs” on a drawing and adjusts their parameters while they draw autonomously. When the user draws circles, squares, and lines, she or he also turns out to cooperate with the software, which instead draws childish faces, childish houses, and twisted lines, respectively, according to adjustable parameters. In addition, the drawing can be manipulated, as in traditional drawing software, by a number of filters and plug-ins, but here these have their own autonomous will. For example, some filters such as “generate instant results” or those called “nq297p” create objects according to quasi-random generative procedures.[30]

 

Figure 4: Adrian Ward’s Signwave Auto-illustrator Image used with permission of Adrian Ward, <www.auto-illustrator.com>.

 

Auto-Illustrator is not radically different from ordinary software, since it uses the standard control interface of the modern GUI. It originated as a parody of Adobe interfaces from Illustrator and Photoshop. Ward explicitly criticizes the radically different interfaces so often seen in digital artworks for being as beside the point: “I want to express myself using the medium of consumer-based application software, which is why Auto-Illustrator doesn’t have a radical interface” (118). Auto-Illustrator follows the standard user interface guidelines, and even its strangest functions appear as tools and filters, like those found in more traditional software. Auto-Illustrator demonstrates an alternative to the way functionality is dressed up as a mere tool in ordinary software. It suggests that functionality includes all kinds of aesthetic and representational choices that are normally hidden under the tool metaphor. It demonstrates the representational character of the instrumental, and that the artist working with specific software has severely limited his or her potential for expression, but it also points out that mainstream software is limited in its potential for creativity because it has to stay within the range of the tool metaphor.

 

Auto-Illustrator paves the way for a discussion of the ways the tool works as a representational and aesthetic structure, a discussion that potentially reconfigures the whole field around artist, tool, and artwork. As Ward expresses it, “if a plug-in can be authored that reproduces that trendy graphical effect that everybody loves, how does this impact upon the role of the graphic designer, and thus how does one treat the software that produced the design? Who is the designer? The user, or the author of the code?”[31] In this way, Auto-Illustrator questions the implied perceptions in ordinary software of the user as the active artist/author and of the software itself as a passive tool. In digital art, specific software and software-hardware combinations play an enormous role–imagine digital photography without Photoshop, electronic music without Steinberg’s Cubase or the products of Roland, hyperfiction without Eastgate’s Storyspace, or multimedia without Hypercard, Director, or Flash. This software conceptualizes the user and the material with which users work.[32]

 

Auto-Illustrator challenges the traditional conceptualizations of user, object, and software. The user is normally conceptualized as active and in control, but could just as well–and perhaps often more in line with the actual truth–be conceptualized as reactive and partly controlled, as in many computer games, where the interface is more opaque and difficult to control. The material with which the user works is most often metaphorized as something well-known and relatively stable–for instance, paper, a photograph, a canvas–but could also be more dynamic, algorithmic, cybernetic, and emergent, as in Auto-Illustrator or browser art such as The Web Stalker or Feed. And, finally, the software, which often stages itself as passive, supportive, and neutral, could instead be active, interfering, and creative. This may be what qualifies Auto-Illustrator as an artwork.

 

In the “preferences” palette, one can find the very button that best expresses the intention of Auto-Illustrator. The “preferences” palette is perhaps the most sacred place on the interface (apart from the “deep level” of code which is “behind” the interface and thus normally not visible). Through the “preferences,” “settings,” or “control panels,” it is possible to manipulate the very staging of the interface–its colors, language, interaction menus, file handling, warning messages, and so on. It is here that one is allowed to personalize the software and change passwords, here that the author or producers of the software to some extent make the staging of the interface explicit. The preferences section is a small peephole into the backstage area, but it is also where the relations between author, software, and user are defined, though of course within strict limits. It is not possible to change everything, and often one cannot find the setting that controls an annoying feature one wants to change.

 

In Auto-Illustrator, apart from actually choosing “annoying mode” or choosing to “mess up palettes” in the “interface” section of the “preferences” palette, it is possible to enter the “psychosis” section, where the “Important–Don’t push this button” button is located. Here Auto-Illustrator addresses the very ontology of software interfaces. By choosing “Psychosis” the user is already well out of line, and, in fact, the “Don’t push this button” button highlights the normally repressed schism between the functional-instrumental and the mediated in Nake’s definition. What is often a repression of the representational and mediated nature in traditional software interfaces is turned into an expressive psychosis here.

 

Figure 5: Preferences Palette in Signwave Auto-Illustrator Image used with permission of Adrian Ward, <www.auto-illustrator.com>.

 

Focusing on this button helps clarify the schism of the instrumental medium. A button indicates a functional control, by means of which something well defined and predictable will happen; the fact that it is often rendered in 3D simulates a physical, mechanical cause-and-effect relationship. Of course, we know that the button is in fact a symbolic representation and as such a mediation of a functional expression, but we nevertheless see and interpret it as something that triggers a function. It is a software simulation of a function, but normally this simulation does not point toward its representational character, but acts as if the function were natural or mechanical in a straight cause-and-effect relation. Yet it is anything but this: it is conventional, coded, arbitrary, and representational, and as such also related to the cultural. As manufacturers of technological consumer goods from cars and hi-fi equipment to computer hardware and software know, buttons have aesthetic qualities and respond to a desire to push them. This could be a desire for control, but perhaps it is also and primarily a tactical desire. Buttons and knobs are supposed to feel good–one can even read car reviews that criticize the buttons and controls of cars for their cheap “plastic” feel. In Auto-Illustrator this button’s desire to be pushed is paradoxically heightened and pinpointed by the text “Don’t push this button.” One could say that by its apparent denial of functional purpose the button tempts our desire for the functional experience of tactical control and mastery–a strong ingredient in the aesthetics of the interface, even when denied.

 

When the button is pushed, functional realism is cancelled and the interface is itself turned into something that resembles an opaque net.artwork. Text in palettes is turned into gibberish code, windows are moving and sounds are beeping, and the clear distinction between the data and the program, which is necessary in order to sustain the illusion of functional software, is cancelled.[33] This button delineates a borderline between the functional interface and modernist representational aesthetics. Here a new critical, functional art arises within the instrumental medium of the computer–not as an optimistic compromise but as a rupture between two clashing modes: the representational mode of art and the functional interface. When pushed, the rupture is radicalized, the schism is turned into sheer psychosis, the functional into pure art. Consequently, one should not push this button, because it destroys the delicate balance between the functional and the representational in Auto-Illustrator, between seeing its interface as a tool and as a “pure” representational art form.[34]

 

What are the aesthetical, cultural, and ideological ramifications of functional aesthetics? How does it influence our experience and perception, and what does it do to the arts?[35] The very object of the artwork is once again questioned by such artistic experiments. How does the artwork actually work in a political and cultural economy? The myth of the autonomous work of art is increasingly difficult to sustain–a fact which also makes digital art, net.art and software art somewhat difficult to discuss and sell, though Auto-Illustrator has chosen to use the contested model of proprietary software, and sells licenses to the software.[36] Functional realism as demonstrated by Auto-Illustrator critically highlights central issues in the GUI and in the computer as an instrumental medium. It is about functional changes in the media, aesthetics, and the arts–and about changes in the very concept of functionality itself.

 

The Realisms of the Interface

 

I have maintained that the interface should be regarded as a cultural and aesthetic form with its own art–digital art that goes under the names of net.art, software art, computer games–but I could have included electronic literature that deals with relations between the code and the interface or between the work, the text and the network. A good example of the former is <www.0100101110101101.org>, of the latter Christophe Bruno’s <www.iterature.com>. In addition, questions about cultural aspects of the interface are also raised in contemporary techno-culture and techno music, notably in genres like laptop music. For example, the musician Markus Popp argues that we should understand music as software, in terms of the software processes used to produce it (Popp).

 

Furthermore, I have discussed contemporary realism, since realism is important to the industry, HCI research, and broad cultural conceptions of the computer. I have pointed out that there is more than one realism, that realism is not only naïve illusionism, nor can it be reduced to What You See Is What You Get. As I have argued, one actually sees different things through these different perspectives, although they are clearly related and cannot be fully isolated. Furthermore, my three categories of realism aim beyond the safe borders of the autonomous artwork: illusionistic realism aims beyond pure representation toward immersive simulation, media realism beyond the visual surface toward the imperceptible and unreadable code, and functional realism beyond the artwork as self-contained and disinterested toward a functional aesthetics of the instrumental medium.

 

Realism is ultimately about seeing and reaching reality–a reality that is not something alien “out there” but that consists of media and to a certain extent is constructed with the use of media. This construction of reality through media is both conceptual, as when media functions as models of understanding reality, and directly physical, as when media becomes embedded in the infrastructure of postmodern reality.[37] In its most stringent form, realism puts the very concepts of art and the real at risk simultaneously, so it is encouraging that some of the heated discussions of computer games, net.art and software art have concerned how–or if–one should look at this as art. What are the boundaries of net.art and software art? This discussion has been given impetus by the awarding in 1999 of the Ars Electronica’s Golden Nica prize to the open source operating system Linux in the category of net.art.[38] But when dealing with the interface from a realistic perspective, perhaps the most pressing questions are where the interface is, what it looks like, and how and what it makes us see. Is the interface a moving target we are hunting down in dark corridors of first-person shooters? Is it the opaque digital materiality of Jodi’s game modifications? Or is it hidden in the playful interaction of Auto-Illustrator?

 

Part of the answer is that the interface is a continually developing forms for functional and artistic practices, a moving target artists will keep following and exploring, just as artists have explored the canvas or writers the codex. The materiality of the interface is gradually being rendered visible, readable, audible, navigable, and so forth. Interface aesthetics contributes most to the development of the interface by making sure that it does not become “invisible,” transparent, “subservient to the task,” as Don Norman claimed in 1990, that it does “get in the way” and is explored as a form, a language, an aesthetics. Interfaces are made for interaction and thus keep adjusting to accommodate actual users and uses. For this reason we can never fully grasp the interface as a form, but are compelled to pursue its various and ever-changing appearances.

 

Notes

 

This article is the product of discussions and research carried out with Lars Kiel Bertelsen and Olav W. Bertelsen and has benefited from discussions at the occasions where it has been presented. Stacey M. Cozart has helped with linguistic corrections. Thanks also for extremely valuable comments and corrections from reviewers at PMC, which have helped improve this article.

 

1. In visual arts, questions concerning the art institution and its connections to museology are important, while from a literary perspective questions concerning writing versus code and the functions of authors and readers are important. For the museology perspective, see Dietz. For the literary perspective, see Cramer, “Free Software as Collaborative Text” and “Software Art.” I have explored the literary perspective in several Danish articles and in “Writing the Scripted Spaces.” Within the field of computer games, heated discussions have arisen around the relevance of a narratological approach versus an emerging ludological approach to computer games, the latter aiming at establishing the computer game as an art form of its own.

 

2. There are many combinations of analogue and digital techniques, where parts of the work–its production, storage, distribution, and modes of reception–still take analogue forms, even though other parts are digitized. Furthermore, and perhaps more importantly, differences exist between the ways digital techniques have influenced the ontology and cultural economy of artworks, even within a particular art form. Consider for example the differences between commercial pop/rock and techno music: the digital has only slowly led to substantial changes in most mainstream musical forms, whereas in techno and dj-culture the very ontology of the musical work has changed.

 

3. The development of hypertext and hypermedia should be mentioned, notably the work of Vannevar Bush and Ted Nelson. See Wardrip-Fruin and Montfort, which collects many of the key historical texts.

 

4. Madsen has used this quotation in her interesting attempt to consider the interface as an art form.

 

5. See, for instance, Laurel.

 

6. See, for instance, Macintosh Human Interface Guidelines, chapter 1, on Direct Manipulation, WYSIWYG and metaphors; a new version for Aqua can be found at <http://developer.apple.com/techpubs/macosx/Essentials/AquaHIGuidelines/>. See also Schneidermann, 485-498, in Wardrip-Fruin and Montfort.

 

7. See e.g. Bertelsen and Pold.

 

8. A current example is Switch 18, “Interface: Software as Cultural Production”; see <http://switch.sjsu.edu/nextswitch/switch_engine/front/front.php?cat=44>.

 

9. Latour and Hermant theorize and document how modern urban reality becomes increasingly invisible, though it is continuously visualized.

 

10. See, for instance, “Panoramic Realism,” my article on the realism of Honoré de Balzac. Johnson compares the interface as a cognitive map to “the great metropolitan narratives of the nineteenth-century novel,” though he mainly compares it with Dickens (18).

 

11. Cf. both Manovich and Elkins.

 

12. Johnson is referring to the interface as a postmodern media form which inherits characteristics such as intertextuality, eclecticism, and fragmentation, but the interface is also variously opposed to general notions of the postmodern. At least it suggests a strong technological, functional, and instrumental dimension of the general postmodern culture. Still, it is interesting to re-read one of the defining articles of the postmodern, Fredric Jameson’s “Postmodernism, or, the Cultural Logic of Late Capitalism,” in light of the interface. Jameson hesitatingly illustrates the postmodern by “the distorting and fragmenting reflexions of one enormous glass surface to the other” with explicit reference to the dominant role of computers and reproduction in postmodernity (79). The year Jameson published his article (1984), Apple released its first Mac OS. Perhaps this “convergence of contemporary critical theory and technology”–to repeat the subtitle of George P. Landow’s Hypertext–offers the possibility to update and reconfigure our notions of the postmodern.

 

13. The dichotomy of immediacy and hypermediacy was set up and developed by Bolter and Grusin (1999). See 78-84 for a summary of the criticism of linear perspective in the tradition deriving from Panofsky.

 

14. Max Payne was a huge success for the game development company Remedy, based in Espoo, Finland. A sequel, Max Payne 2: The Fall of Max Payne, was released in 2003.

 

15. On tactical and strategic ways of being in space, see de Certeau, especially XLIV ff. and 139 ff.

 

16. See, for instance, Aarseth, Juul, and Ryan. Aarseth and Juul are leading opponents of the general academic trend of discussing games as narratives. Instead they emphasize the ludic ontology of games, pointing to the importance of game-play, choice, and ergodics and quests. While they have a point in criticizing narrative hegemony as it is carried out, for instance, in the ideal of “interactive fiction/story” and in many narratological game analyses, Ryan articulates a more balanced (though also critical) version of narrative traits in digital media.

 

17. The game is explicit about this construction. The internal first-person narrator, Max Payne, states that “this is how it ended,” and later says that “to make any kind of sense of it I need to go back three years . . . Back to when the pain started.”

 

18. This is especially explicit in Chandler’s The Little Sister.

 

19. In addition, the game interface itself remediates other media to a large extent, and it is thus a perfect example of Manovich’s cultural interface category (62-115)–the plot is largely told through a photo realistic graphic novel complete with frames and text bubbles. The game play is, as in many other computer games, interrupted by cinematic sequences using montage and spectacular camera movements, but the game play is itself inherently cinematic in nature. Basically one sees the scenery through a camera behind and above the avatar, but there are also cinematic effects that can be enabled by the player. In particular, the “bullet-time” and “shoot-dodging” slow-motion effects are important for the player’s tactics.

 

20. An exhibition complete with a printed catalogue, in which Cramer argues that “if the contemporary art system were not fixated on displays–whether of opulent visuals or of political correctness–and on material objects to be sold, Jodi might be recognized as the most important artists of our time” (Baumgärtel and Büro).

 

21. See, for example, the net.artworks at <http://404.jodi.org> and <http://oss.jodi.org>, especially the executable files that can be downloaded.

 

22. Sod is a modification of Wolfenstein 3D; it is available from <http://sod.jodi.org>.

 

23. On the culture of making new levels of and versions or modifications to an original game engine, see Trippi and Huhtamo.

 

24. It is not entirely new to claim a connection between representation and outward action in realism and realistic theories such as speech-act theory and some aspects of dialectic materialism and so on.

 

25. Heidegger distinguished between objects that are “ready to hand” and “present at hand,” the former functioning seamlessly as tools (like a hammer) withdrawn into the activities in which one is engaged–a distinction that has played an important role in HCI through Winograd and Flores’s introduction in 1986 (cf. Dourish).

 

26. The “Software” exhibition is documented with a reprint of the original catalogue in Wardrip-Fruin and Montfort, 247-257.

 

27. Cramer follows Henry Flynt in arguing that “concept art is a kind of art of which the material is language,” and consequently that “software can be seen and read as literature.”

 

28. The Web Stalker can be downloaded from <http://bak.spc.org/iod/>. Other examples of browser artworks are Mark Napier’s Feed, Riot and Shredder (<http://potatoland.org/>), and Maciej Wisniewski’s Netomat (<http://www.netomat.net/>).

 

29. Auto-Illustrator won the Transmediale 2001 price for Artistic Software and received an honorary mention at Ars Electronica in 2001. At the time of writing, the current version is 1.2 (Windows).

 

30. Filters work through plug-ins that can be written and installed independently by the user or by a third-party producer.

 

31. See “About Auto-Illustrator” in the documentation that comes with Auto-Illustrator 1.2. For example, in Adobe Photoshop there are filters that more or less automatically create painterly effects or glass mosaics from a given image. Auto-Illustrator also demonstratively criticizes the pointlessness of many of the automated effects. A good example is the mocking “stupid and pointless” filter in Auto-Illustrator, which in fact does nothing at all and takes a long time doing it.

 

32. See Albert, who discusses the author function in Auto-Illustrator. He also discusses the way traditional net.art interposes between media producers and media consumers, while software art interposes between software producers and media producers, thus higher up in the chain of production: “the most influential position is clearly that of the software programmer, and the most obvious point for intervention is there, between the software producer and the media producer.”

 

33. Other net.artworks that reflect this aesthetic include works by Jodi, especially the OSS series of software that can be downloaded from <oss.jodi.org>.

 

34. See also Albert’s re-phrasing of Mathew Fuller’s “not-just-art” concept, a concept Fuller developed for I/O/D’s Web Stalker, and that Albert argues is more fitting to Auto-Illustrator. Fuller writes about Web Stalker as not-just-art in the sense that “it can only come into occurrence by being not just itself. It has to be used” (43).

 

35. When my five-year-old son sees an image on a computer screen he clicks on it like a maniac in order to enter or execute it. By default he sees digital images as interfaces with hidden links and functionality in addition to seeing them as representations.

 

36. Even here it uses the medium of consumer-based application software, as Ward has pointed out, and not a radical alternative such as the open source model, though parts of Auto-Illustrator such as the plug-ins are in fact quite open and invite co-authorship.

 

37. Embedded computers and media are currently discussed within the research fields of pervasive computing, augmented reality, mobile handheld devices, etc. However, our urban environment is not media saturated exclusively because of future technologies. See e.g. Latour and Hermant.

 

38. Among others, the net.artist and software art curator Alexei Shulgin has argued against seeing Linux as art, claiming that it is functional software while net.art is non-functional, an argument Cramer rightly rejects as “late-romanticist” in “Free Software as Collaborative Text.” See also the discussion on the Nettime Mailing Lists after the prize was awarded under the thread “Linux wins Prix Ars due to MICROSOFT INTERVENTION.” The pressing question seems to be whether art can embrace the functional dimensions of software or whether art has to be dysfunctional in order to maintain a critical distance.

 

Works Cited

 

• Aarseth, Espen. “Beyond the Frontier: Quest Games as Post-Narrative Discourse.” Narrative Across Media. Ed. Marie-Laure Ryan. Lincoln: U of Nebraska P, 2004.
• Albert, Saul. “Useless Utilities.” < http://twenteenthcentury.com/saul/useless.htm>. Rpt. in Ward 89-99.
• Apple Computer. Macintosh Human Interface Guidelines. New York: Addison-Wesley, 1992.
• Baumgärtel, Tilman, and Friedrich Büro. Install.exe–Jodi. Basel: Christoph Merian Verlag, 2002.
• Bertelsen, Olav W., and Søren Pold. “Criticism as an Approach to Interface Aesthetics.” Proceedings of the Third Nordic Conference on Human-Computer Interaction. New York: ACM, 2004. <http://doi.acm.org/10.1145/1028014.1028018>.
• Bolter, Jay David, and Richard Grusin. Remediation–Understanding New Media. Cambridge: MIT P, 1999.
• Chandler, Raymond. The Little Sister. 1949. New York: Vintage, 1988.
• Cramer, Florian. “Concepts, Notations, Software, Art.” < http://userpage.fu-berlin.de/~cantsin/homepage/writings/software_art/concept_notations//concepts_notations_software_ar t.html>.
• —. “Free Software As Collaborative Text.” <http://userpage.fu-berlin.de/~cantsin/homepage/writings/copyleft/free_software/free_software_as_text/en//free_s oftware_as_text.html>.
• de Certeau, Michel. L’invention au quotidien: 1. arts de faire. Paris: Gallimard, 1990.
• deLahunta, Scott. Lecture, Aarhus University, 1999.
• Dietz, Steve. “Curating (on) the Web.” Museums and the Web: An International Conference. Toronto, Canada. 22-25 April 1998.<http://www.archimuse.com/mw98/papers/dietz/dietz_curatingtheweb.html>.
• Dourish, Paul. “Seeking a Foundation for Context-Aware Computing.” Human-Computer Interaction 16 (2001): 2-3.
• Elkins, James. The Domain of Images. Ithaca: Cornell UP, 1999.
• Fuller, Matthew. “A Means of Mutation.” Readme! Filtered by Nettime. Eds. Josephine Bosma, Pauline van Mourik Broekman, et. al. New York: Autonomedia, 1999. 37-45.
• Huhtamo, Erkki. “Game Patch–The Son of Scratch.” Switch 12 (July 1999). <http://switch.sjsu.edu/nextswitch/switch_engine/front/front.php?artc=119>.
• Jameson, Fredric. “Postmodernism, or, The Cultural Logic of Late Capitalism.” New Left Review 146 (Jul-Aug 1984): 53-93.
• Johnson, Steven. Interface Culture: How New Technology Transforms the Way We Create and Communicate. San Francisco: Harper, 1997.
• Juul, Jesper. “Games Telling Stories? A Brief Note on Games and Narratives.” Game Studies 1.1 (July 2001) <http://www.gamestudies.org/0101/juul-gts/>.
• Klein, Norman M. The History of Forgetting: Los Angeles and the Erasure of Memory. New York: Verso, 1997.
• Landow, George P. Hypertext. Baltimore: Johns Hopkins UP, 1992.
• Latour, Bruno, and Emilie Hermant. Paris ville invisible. Paris: La Découverte, 1998.
• Laurel, Brenda K. “Interface as Mimesis.” User-Centered System Design: New Perspectives on Human-Computer Interaction. Eds. Donald A. Norman and Stephen W. Draper. Hillsdale: Erlbaum, 1986. 67-86.
• “Linux Wins Prix Ars due to MICROSOFT INTERVENTION.” Online discussion. 6 Sep. 1999. Nettime Mailing Lists. <http://amsterdam.nettime.org/Lists-Archives/nettime-l-9909/msg00038.html>.
• Madsen, Iben Bredahl. “Interfacet som kunstform.” Diss., University of Aarhus, 2002.
• Manovich, Lev. The Language of New Media. Cambridge: MIT P, 2001.
• Max Payne. Remedy/3D Realms, 2001.
• Murray, Janet. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Cambridge: MIT P, 1998.
• Nake, Frieder. “Der Computer als Automat, Werkzeug und Medium und unser Verhältnis zu ihm.” Menschenbild und Computer. Selbstverständnis und Selbstbehauptung des Menschen im Zeitalter der Rechner. H. Buddemeier (Hrsg.) Bremen: Universität Bremen, Medienkritische Reihe 3, 2000. 73-90.
• Norman, Donald A. “Why Interfaces Don’t Work.” The Art of Human-Computer Interface Design. Ed. Brenda Laurel. Reading: Addison-Wesley, 1990. 209-220.
• Pold, Søren. “Panoramic Realism: An Early and Illustrative Passage from Urban Space to Media Space in Honoré de Balzac’s Parisian Novels, Ferragus and Le Père Goriot.” Nineteenth Century French Studies 29 (2000-2001): 47-63.
• —. “Writing the Scripted Spaces–Understanding Writing In A Digital Context.” Unpublished essay, 1998. <http://imv.au.dk/~pold/publikat/scripted.pdf>.
• Popp, Markus. Interview. “Markus Popp: Music as Software.” By Sam Inglis. Sounds on Sound Oct. 2002 <http://www.sospubs.co.uk/sos/oct02/articles/oval.asp>.
• Ryan, Marie-Laure. “Beyond Myth and Metaphor: The Case of Narrative in Digital Media.” Game Studies1.1 (July 2001) <http://www.gamestudies.org/0101/ryan>.
• Trippi, Laura. “Deep Patch.” Switch 12 (July 1999) <http://switch.sjsu.edu/CrackingtheMaze/laura.html>.
• Ward, Adrian, ed. Auto-Illustrator User’s Guide. Plymouth: Liquid/Spacex, 2002-2003.
• —. Auto-Illustrator 1.2. Signwave, 2003. <http://www.auto-illustrator.com/>
• Wardrip-Fruin, Noah, and Nick Montfort, eds. The New Media Reader. Cambridge: MIT P, 2003.