When our music device changed from component stereo to WALKMAN, a lot of people began to take the music, which had always been listened at home, outside.
That is to say, our society shifted so that people ‘wear’ music.
VR is a new media that allows people to wear space.
That’s the reason why VR gets along sound very well without discomforting viewers.
Then, what should we do to render the ‘story’ and ‘groove’ created by sound as a form optimized for 3D instead of 2D?
An answer to that question is ‘ENCLOSURE’, a VR work created by ‘TeamMIKAMI’ that consists of the members from OMNIBUS JAPAN.
This work won the Kaleidoscope prize at NEWVIEW AWARD 2018.
It is an installation-art work with the concept of ‘the space of sound’.
In this article, I asked about how they decided to create this work, the aim and various techniques applied to this VR work.
Profile
TeamMIKAMI (from OMNIBUS JAPAN) | (@TeamMikami) Creative unit | Japan The title: ENCLOSURE
A creative unit consisting of three members of Omnibus Japan, Shozaburo Oka, Yuuki Kawakami and Hideki Mikami. They actively works on the art works for various devices, such as a Geo-Cosmos content ‘inside’ in National Museum of Emerging Science and Innovation, an installation art ‘NOCTILUCA’. For this VR work, Keisuke Tsukamoto joined them as Sound Designer and Natsuo Nishi joined as CG Designer.
It’s not BGM – Enter the space of sound
— How did you know about NEWVIEW?
Kawakami: Since I got information about NEWVIEW workshop from my supervisor in the company and I attended it.
— What made you decide to apply for it?
Kawakami: Since I thought that our participation in this award could be a good opportunity to show our work to a lot of people including the judges. … It’s also true that we were lured by the prize for Grand Prix award, I could say.
ENCLOSURE from STYLY on Vimeo.
— I see. What concept is behind ‘ENCLOSURE’ you created this time?
Kawakami: We created it with the concept of ‘The space of sound’. When starting to create this VR work, we wanted to make something related to the picture and sound like an audiovisual art. Instead of using music as BGM in space, we imagined that people could enter the space of music through HMD.
— How did you create this VR work and how long did it take?
Mikami: It took a month to come up with the idea and test it. The actual production took about half a month. We used Cinema4D for animation and put sound and visual together on Unity. We created the music with Cubase. 
Mikami: Regarding the actual procedure of the production, we firstly found it very important how to synchronize sound with pictures when making ‘ENCLOSE’, a work focusing on the relationship sound and video. If you import the FBX with animation and the sound file with the same duration into STYLY separately, the animation gradually deviates from the sound as it loops. We tried various ways and we found that it synchronizes by exporting the object and sound together from Unity to STYLY. Regarding spatial audio, we found it possible to reproduce an object-based spatial sound when viewing some works in STYLY VR and came up with the direction to create a surrounding sound field. 
Next, we faced a problem of how many objects we can place in the space. To answer it, we could do nothing but repeat steady work, trying to import a lot of objects and reducing them when they didn’t work. Regarding this point, it was very helpful that the Unity plug-in became available in STYLY and STYLY was updated to import the whole scene from Unity, during this contest. (Reference: How to upload an asset from Unity to STYLY ） In addition, we placed the mono sound file at each object to make sound accordingly, set the audible range for each object and turned off the Doppler effect, which is activated by default, not to change the pitch when you are moving away. This time, thanks to Unity plug-in in STYLY, we completed the work without writing any code. 
— Thank you. Could you tell us about the technical aspect on overall 3D expression, if you have any?
Oka: It’s very interesting that I can control the direction and the volume of sound interactively by assigning a sound to each object. We wanted to apply this function to the work at the initial stage of the production. However, it’s difficult to control, so we couldn’t get the result we wanted. For example, the sound cracked in the situation when multiple objects overlapped. Later, for some sound effects, we solved the problem by mixing down the sound effect and the song together into one WAV file. After the mix-down process, we published the space to STYLY and finished it up by adjusting the panning, sound quality, volume and so on repeatedly while checking the pictures through HMD.
Blend ‘What you can see’ and ‘What you can hear’
— What did you care about in this VR work?
Oka: When creating objects with Cinema4D, I really cared about how naturally viewers can feel the combination of ‘what they can see’ and ‘what they can hear’. Then, we came to the conclusion that one of the most suitable expression is realistic animation by physical calculation. We carried out the physical calculation on CINEMA4D. And, after baking the animation, we converted it into the FBX file and exported it to STYLY.
Tsukamoto: The most difficult point was to balance the volume and property of VR-dynamic sound against those of the fixed-position sound. By repeating the sequence of making the sound and matching the video with it, we polished up the sound-video matching until we were satisfied with. We used Cubase for DAW. 
— Finally, could you tell me about your future plan?
Mikami: As TeamMIKAMI, I would like to produce various contents. It’s really meaningful to create by a team, not a single person, and by a small number of people, not a lot of people. In this way, each person can produce work with his or her speciality while making a decision. Through this approach, we want to create something that left in someone’s memory as a primitive emotion such as ‘interesting’, ‘beautiful’, which leads us to a new ‘craftsmanship’. ‘Sound’ is one of the important elements to increase the presence of 3D space. How do you utilize that element for art, music and lifestyle? Of course, there is no answer yet. But, because we don’t have the answer, it is important to have more artists who experiment with various approaches now. I will be glad to see that more works with the theme of ‘VR x Sound’ are released through STYLY.