Lytro and the Medium of the Interactive Image

Digital Technologies, Further Mediums Add Comment

When a new technology is being released to the market, how can you tell if it is just another gadget or it represents a fundamental paradigm shift? Lytro is a recent technology which, at least initially, seems like it might be the latter. Whether it becomes a commercial success or not is dependent on many other factors, but looking at it from the point of view of The Virtual Space Theory reveals that it is more than just a technology – rather, it introduces and reinforces an altogether alternative understanding of what images can be.

Lytro is a type of camera that promises to take pictures which can be focused later on when viewing them: If you want to sharpen an element in the foreground, click on it and it becomes focused; If it is a background detail that you want to see, click on it and now the focus of the image is again set accordingly. Since this technology is new, it is not yet available for demonstration from within this blog, but I recommend that you try it for yourself here.

How is this done? It uses a novel lens system which is fitted into a specialized handheld camera. This complex set of lenses replaces the need to set the focus of the image before taking it, and actually captures the whole depth of the visual field – or in other words, it captures all of the potential focal points of an image. All this information is stored in a digital file, and as long as it is uploaded to a website that is equipped with its image viewer, anyone can adapt the focus of the images as they wish – upon viewing them.

What is so special about it? What it provides is not just another way of producing photographs or of viewing them (like a digital camera does), nor is it a tool for manipulating photographs that exist independently of it (like Photoshop is). Rather, it ventures away from photography into the territory of a whole other pictorial medium. With its combination of image capturing device, digital file format, and image viewing software, the overall result can simply no longer be considered to be a static image, but rather an interactive image. The viewed place is still static, but the image of it is not – it is adjustable by the viewer.

As proposed in my book The Architecture of Virtual Space, all pictorial images are windows to virtual space, regardless of their medium. What does tell mediums apart is the type of window that they provide, e.g., photography provides a static window, and film provides a dynamic window. Thus, a window to virtual space which inherently allows the adjustment of what one sees through that window is by definition another type of window – that is, another medium altogether.

In that sense, Lytro is reminiscent of previous technologies that have expanded photography beyond static images and provided various aspects of what interactive images can be. Two such comparable technologies are QuickTime VR and HDR images:

- QuickTime VR is a means of taking photographs that capture all views around the camera in 360 degrees – around, above, and below. Stored in a specialized image file, these images can be viewed in QuickTime player, and by clicking and dragging the mouse the viewer can change his angle of view of the place seen through the image. (To see it for yourself you can find the QuickTime viewer here, and sample images for it here.)

- HDR (High Dynamic Range) images are a technology designed to capture and represent a wide range of light exposures in a single image file. This is very useful for many aspects of digital imaging, but it also provides the ability to choose the degree of light exposure of an image while it is being viewed, rather than when it is being produced. (To see it for yourself you can find an HDR viewer here, and sample HDR images here.)

Therefore, Lytro joins QuickTime VR and HDR images in providing experiences of virtual space in which the viewer has a degree of control over what he sees. Each of them explores yet another variation on the medium of the interactive image – whether by allowing the viewer to choose his angle of view, the light exposure, or the focal point. Over time we may even see a technology that allows all three, as well as additional forms of image control.

However, these should not be seen only as enhanced forms of photography. After all, these same types of images can be produced by 3D computer graphic programs just as well. Thus, the fundamental development here is not only the impressive technique of making images, but in the widening understanding of what pictorial images can be: virtual places that can be actively explored in ever more ways.

Is Theater Virtual?

Further Mediums, Terminology Add Comment
Part 3 of 3 in the series Is X-Y-Z Virtual?

The experience of theater is quite unique: You enter a hall filled with seats facing a curtain over a stage, and at a certain moment the lights go out, the curtain rises, and a seemingly other world is revealed – lying physically right there in front of you. What is the nature of this ‘other world’? Does the term ‘virtual’ capture it? If it does, how exactly, and if not, why not? While such questions have surely been addressed from the perspective of theater theory, this blog post rather discusses them from the point of view of The Virtual Space Theory.

It might initially seem natural to describe the phenomenon of theater as ‘virtual’, but this is mainly due to the multiple inconsistent uses of this term as discussed in a previous post. And although it might still make sense metaphorically, this term does not fully capture the essence of the issue. The key to realizing this is to ask ourselves: where is what we experience in a theater?

To begin with, according to The Virtual Space Theory, visual experiences can only take place in one of three types of space: physical, mental, and virtual. Physical space is the physical world in which our bodies and their surroundings are located; mental space is our mind’s eye, where we replay visual memories and construct visual ideas; and virtual space is the space that is seen through pictorial images of any kind. The exact distinction between the three as well as this particular definition of virtual space are fully elaborated in my book “The Architecture of Virtual Space” and in the article derived from it.

From this point of view, then, in which of these three spaces does theater occur? The answer is simple and straightforward: in physical space. Since the stage is physical, the sets are physical, and the actors are physical – the phenomenon of theater occurs in physical space. To understand theater, therefore, is to understand what exactly happens in the physical world during the performance of a theater play. This is probably best accomplished by comparing the physical stage before and after the curtain rises:

For example, let’s take a theatrical performance of the ancient Greek play Oedipus Rex by Sophocles. If we were to walk onto the stage and peek behind the curtain just before the performance begins, what might we find? A physical stage that is within the physical building of the particular theater we are visiting; various props and constructions made of wood or plaster, covered in paint so as to appear to be stone, arranged such that they seem like part of the ancient Greek city of Thebes; and actors wearing make-up and costumes soon after having worn jeans and talking on a cell phone just a little while earlier. And yet, once the lights go out and the curtain rises, the physical space of the stage and everything on it is totally transformed: What now lies before us is the ancient city of Thebes, it is built of stone, and the people walking in it are living and talking ancient Greeks. Furthermore, it is precisely because this is so that we even go to the theater.

What exactly is the nature of this transformation? Since it is all happening in the physical world, the term ‘virtual’ does not quite capture it. Alternatively, to say that it’s all in our mind does not account for what is providing such an experience in the first place. Rather, what is happening here is a simulation: Within the socially agreed convention of the time and space of a theatrical performance, the stage, sets, and actors function as physical simulacra (or ‘objects that perform a simulation’). Concertedly, they transform from mere stage, sets, and actors to become ancient Thebes, buildings, kings and queens. Yet there is nothing virtual about it.

This, however, does not mean that virtual space can never participate in the formation of such a theatrical experience – quite the contrary, it frequently does. To create the simulation of a (factual or fictional) physical place on a theater stage, a theatrical production is not limited only to props and physical constructions, but may also employ pictorial images:

Often made on a large scale and strategically positioned on the stage, the virtual places that are seen through such images can be made to appear to be direct extensions of the physical space of the stage. In such cases, the virtual places and virtual objects that are seen through these images function as virtual simulacra.

When such images are placed alongside the physical props that are located on the stage, virtual and physical simulacra seamlessly merge in the creation of a continuous visible world – a simulation of another time and place enacted for us in the physical space of the theater hall at the present moment.

Are Light Projections Virtual?

Further Mediums, Painting Add Comment
Part 2 of 3 in the series Is X-Y-Z Virtual?

The winter holiday season is also a celebration of lights, and many cities pride themselves on the beautiful light installations along their streets and on their buildings. One such type of light installation takes the form of the projection of images onto physical buildings. There are many examples of such light shows, which vary in quality as well as in their degree of architectural emphasis. Here is one recent example from the city of Lyon’s festival of lights:

Another example is the following:

Probably one of the most suitable examples for this discussion, however, is the now famous light projection presented last summer in Kharkov, Ukraine:

Such light projections raise questions regarding both the nature of their visual content as well as their interaction with the buildings onto which they are projected. What is the actual location of what is seen in them? What is the role of the physical building? What happens to it when it receives the projections? This post, therefore, addresses these questions from the point of view of The Virtual Space Theory.

Since light projections can change in time different segments of a projection may employ different types of images altogether. In this case, some segments present abstract patterns of light onto the building, some segments present pictorial images onto it, and some actually transform and redefine the building itself. The first two types use the building as merely a neutral screen on which to project light patterns that are essentially independent of the building, which could be easily replaced with another building without fundamentally altering the nature of the resulting light show. In such cases, therefore, the term ‘virtual’ applies only to the visual content seen in the pictorial segments of a projection, but not to the light projection itself.

The truly fascinating segments of such light shows, however, are those which directly relate to the architectural structure of the building itself. The visual content of such a light projection is an alternative version of that building – whether deforming, collapsing, or evolving – presented in the exact same physical location as the original physical building, while appearing to replace it. Additionally, the magic of the last example is in its seamless transitions between the different types of images, and their resulting in a single continuous presentation made for a specific physical location.

This type of light projection is probably best understood by comparing it to the ages-old art of ‘trompe l’oeil’ mural paintings. Made already in antiquity and reaching wide production during the Baroque period, these were interior paintings made on walls and ceilings so as to appear to extend the space of the rooms in which they were painted. In the book The Architecture of Virtual Space, I proposed that their key difference compared to regular pictorial paintings is that in addition to creating virtual places, trompe l’oeil mural paintings present their virtual places as if they were a direct extension of the physical location in which they were painted.

For example, Fra Angelico’s Madonna of the Shadows from 1439, with its painted columns and their shadows, gives its whole scene a sense of direct presence in the convent of San Marco where it was painted:

Contemporary uses of trompe l’oeil paintings probably tend be found more on outdoor surfaces, providing added interest to not-so-interesting locations. One example is the work of mural painter John Pugh:

Architectural light projections, therefore, combine the effect of outdoor trompe l’oeil paintings with the added dimension of time. The buildings they create are completely virtual and are located inside of virtual space, yet through their particular way of presentation they make us perceive them as if they were actually located inside of the physical world. This is achieved by two main means. First, the architecture of the created virtual building is designed in full accordance with that of the existing physical building onto which it is projected. Second, the context of the virtual building’s presentation in an already-physical location inevitably adjusts our perception such that we accept what we see as if it too were located in the physical world – right there along with the street, the trees, the wind, and our own physical selves.

In addition, when such a transformation is successful, then even the parts of the show that are regular pictorial projections of a non-architectural nature receive a rare added depth of their own. Normally, the effect of a pictorial image is to make us perceive a virtual place that is located somewhere else – that is, somewhere inside of virtual space. In this case, however, the successful integration of pictorial images with the virtual building makes them appear as if their visual content were actually located within the volume of the virtual building itself. And since the projection of the virtual building onto the physical building already makes the virtual building appear to be located in the physical world, then even the content of the pictorial images now seem to be part of the physical world just as well!

By the use of this cunning device, then, a virtual fish could be made to appear as if it were swimming behind and in front of the façade of a physical building in the physical world… or in other words – as it would seem – a virtual fish in physical space. ;)