Let’s think about the direction of VR by referring to the direction of movies.

Have you ever thought about creating content that is different from movies or photos, and has a unique appeal that only VR can offer?
If you can create a production in VR that has never been seen in any other media, you may be able to create content that will surprise and impress the people who see it.

In this article, I will discuss what kind of content can only be created in VR by comparing three works uploaded to STYLY and a movie, which is a relatively familiar visual media, and think about the expressions that can only be created in VR.

I will not systematically deal with the techniques and grammar of film, but rather discuss the differences between VR works and other media. How to handle motifs will not be discussed in this article because it can be complicated. Instead, I would like to discuss some of the points, mainly camerawork and editing, while touching on actual works and thinking about the direction of VR.

VR works to be discussed in this article

I will consider the direction of the following three works.
All of these works were created with the VR experience in mind.
It may be interesting to briefly look at the following works first, and then look at this article to see from what perspective the techniques are introduced.
Also, the article contains spoilers for some of the works. If you want to avoid spoilers, please try experiencing it in VR before the part explaining the work!

1.PIECE OF STRING

2.Displays XR

3.gaze on me

Thinking about how to tell a story with a camera in VR

It may seem obvious, but movies and photos are basically shot and created by cameras.
Therefore, depending on how and what the camera captures, the narrative of what is shown in the image is created.
In this chapter, we will touch on some examples of how cameras are often used in movies, and think about how to produce them in VR.

Up / Loose

Up and loose (long) are relatively straightforward terms that many people can imagine.
A close-up shot is one that is taken close to the subject, while a loose shot is one that is taken away from the subject.

Close-up shots can be categorized into different types, such as those that capture only the face, the upper chest, or more detailed parts of the body.

In the image on the left, we see a close-up of a person.
This allows us to tell a central story about the personality of a particular subject, such as what gender and age this person is, what kind of facial expression they have, what they are looking at, and if they are talking, how they are talking.
It also shows that this person is the one who should be discussed in detail in this image.

The image on the right, on the other hand, is an image of a loose shot.
It tells us about the age and season of the setting in the image, such as what kind of place it is, what kind of atmosphere it has, what kind of colors it has, and what it represents.
In the case of games, the worldview and setting may also be discussed.

Pan shot

A pan shot is a shooting technique in which the camera is moved in a straight line horizontally or vertically.
For example, after showing a certain person, the camera is slowly moved in the direction of the person who spoke to the camera, or the moving subject is followed by a dolly (a shooting platform with rails and tires).
There are other methods of moving the camera, but by moving the camera in this way, you can explain the stage or create a situation.
For example, in the image above, the camera is moving in the same direction as the people walking along the street to convey the atmosphere of the entire city and the people walking there.
Then, while showing the red building that will lead to the next scene, you can convey the relationship between the city and the building, including the changes in time.
This method can also be used when describing the grandeur of a landscape.

PoV(Point of View)

Point of View is a shooting technique that uses a camera to create an image from the perspective of the characters.

The image on the left is an image of a video shot using PoV.
If the characters looking at this hand are shown in the scene before and after, such as the image on the right, we can see that it was not me looking at the image, but the characters’ eyes that were looking at this hand.
It is also often used to capture the faces of two speakers in a dialogue scene.
It can also be used to show the situation of a surveillance camera image without explaining that the image is from the perspective of the surveillance camera.
This technique allows the viewer to experience the image from the point of view of the camera, and can also be used to depict the psychology of the characters looking at the subject depending on how the subject is photographed.

In the case of VR

In the case of VR, if you want to create a production similar to the shooting techniques described in 1, 2, and 3, you will need to use a different approach and camera.
In VR, basically, what the viewer looks around and sees becomes the subject for the viewer.

As shown in the left image in the above figure, in movies and photos, the image is what the camera sees in a specific direction.
In VR, on the other hand, the subject matter is 360 degrees up, down, left and right from where the viewer is standing, as shown in the image on the right.
If we try to use the above techniques in VR with these characteristics, we can design a VR space as follows.

In the case of a close-up experience in VR that explains the features of a particular object, the user needs to approach the object and look at it carefully.
Therefore, it is possible to create such an effect by creating gimmicks and lines that make the user want to look at the object in detail.
In addition to highlighting what you want the user to see in detail, you can also explain the same object by placing it in a large area.
In the case of Loose, the starting point can be a place where the entire scene can be viewed from a distance.

To show the time and relationship of a specific scene in VR space, like in Pan shot, we can think of creating a gimmick to guide the user’s gaze, for example, the user himself can follow a certain object to advance, like a dolly camera.
To have the viewer take the point of view of other characters, as in PoV, we might consider not only switching the camera itself, but also having the characters talk to the viewer, explaining things through the UI, or changing the viewer’s clothing.
In the case of a character with a distinctive gaze, such as when switching between a human and a monster, changing the height of the viewpoint of the object in the VR scene or changing the scale of the object in the VR scene can make the viewer feel as if they are in the viewpoint of the character.
It is also possible to explain the situation by inserting images into each VR scene, but this may give the viewer the feeling that there are more layers in the VR scene and the sense of immersion is lowered.

In terms of such camerawork, “PIECE OF STRING” achieves a performance unique to VR.

In this work, a room with a sense of life created by high-precision photogrammetry is spread out in front of the viewer.
The viewer can follow the lines in the room to move through it.
As the viewer progresses through the rooms, they are continually presented with rooms that change in scale and direction.

“One second after throwing a pebble into the lake, the pebble disappears and only ripples remain. Similarly, when the people who live in a house disappear, will their reverberations be the only thing there?”

As the text above explains, by continuously looking at an empty room from different angles, scales, and directions, the viewer is given the opportunity to observe what a room is, what life is, what space is, and what it means to observe them.

The viewer’s own body does not change, but the room displayed on the viewer’s VR goggles keeps changing, and in this way the viewer can see the room from the perspective of another body, such as a close-up or loose view, or panning the room itself.

Consider editorial perspectives such as camera switching

In the previous section, I have considered how to capture the subject in the camera.
Next, let’s consider the production in VR from an editing perspective.

In film, there is a term called continuity editing.
Continuity editing refers to editing to create a unity of time and space when multiple shots are lined up to create a single image.
If the camera moves randomly with respect to the subject when connecting the parts of the video, or if the objects reflected in the camera change without any clear causal relationship between them, the viewer will not be able to recognize time.
In order to avoid such problems, the basic principle of film is to edit for uniformity.
In this way, video works can convey psychological changes in the characters or explain situations to the audience by devising ways to create continuity through editing.
This may be easier to understand if you imagine a four-frame comic.

However, VR as of 2021 is generally characterized by the difficulty of editing to switch between multiple scenes.
One reason is that VR scenes are displayed in 360 degrees and have depth with spatiality, so it often takes time for users to look around or walk around to understand a single scene.
Another reason is that due to the hardware of VR devices, it takes some time to load complex scenes, so the more you try to switch between scenes, the more the content can only be experienced on high performance devices.

This is one of the reasons why the more you try to switch between scenes, the more the content can only be experienced on high performance devices.
This is similar to the way of creating a long shot in a movie.
A comparison of the two images is shown below.

The image on the left is about a movie.
For example, it is relatively easy in video editing to show the temporal changes of a human being in three shots.

In the case of VR

In contrast to the above editing of video sequences, in VR, multiple subjects can be lined up in a single scene for a single continuous flow in a single space, as shown in the figure on the right. The blue color is the user and that is the image of the user moving around in the VR space and viewing it in chronological order. However, it is also possible to arrange chronological objects vertically in the depth direction, or to create a space in which the current space and the space before and after it are not so continuous by creating multiple L-shaped bends.

One of the works that create this kind of editing effect unique to VR is “Displays XR“.

In this work, viewers can experience a series of vertical spaces in turn.
In each space, the media and devices in the space wake up, transform, and change, and the room itself shows its face as media.
Such changes are smoothly aligned by a system of moving from one room to the next through a window at the far end of each room.
It is a work that uses not only the media such as the displays in the work, but also the media nature of the space created by VR and the windows that bridge them, both metaphorically and editorially.

Considering the characteristics of the camera itself

Up to this point, I have been dealing with how to capture a subject with a camera, and how to edit the images captured in this way.
I’d like to take this a step further and think about the characteristics of the camera that are unique to VR.
It may seem obvious, but the images that can be captured depend on the characteristics and performance of the camera or lens itself.
This includes telephoto and wide angle, the colors you can capture, and the characteristics of light.
First of all, as a basic premise, the camera used in VR is also designed to reproduce a real camera.
For example, in Unity, you can control the camera functions with the CineMacine function.
By using these functions, it is theoretically possible to reproduce most of the images shot in movies, except for the difficulty of reproducing texture and lighting.

In the case of VR

When considering the characteristics of the camera, one of the most unique characteristics of VR is that the camera is not a physical object.
Cameras in reality have weight, thickness, and hardness. Therefore, it needs a place to lift it, a speed at which it can be moved, and a space that it can enter, which in no small part creates a barrier to where you can take images.

In the case of VR, the camera can be placed in any location.
For example, we can freely place cameras in outer space, underwater, or even inside a person’s body to create a unique VR perspective.
In addition, the cost of shooting such images may vary greatly, and the restrictions on the images that can be created by individuals are often smaller.

One work that makes use of these characteristics of VR is “gaze on me.”

In this work, we first see two people staring at each other, and then the camera goes inside the people.
The camera then goes inside the person, where we can see abstract objects representing the eyeballs, brain, and nerves.
Here, by entering a human’s head, it is revealed that the first scene was that human’s eye (PoV).
In real life, a camera cannot move back and forth between the outside and inside of the gaze, between the human eye and the brain, but this work allows the camera to capture such visuals.
It can be said that this work deals with the intersection of the human gaze and the gaze of the viewer at the same time.

In this article, we have been thinking about how VR can handle the grammar that has been created in other media such as film.
I hope that this will give you a perspective on how VR and AR are being used to produce other works than the ones I have dealt with.
Also, when you create VR inspired by various media other than VR, such as film and photography, please think about how you can direct it.