(JP) バーチャルモーションキャプチャーを使ってVtuberになる方法

In this article, I introduce an application called VirtualMotionCapture.
VirtualMotionCapture is the application to control 3D model in VRM format by VR devices and I recommend it especially for those who want to become Vtuber.tuber.
And, since it can be launched simultaneously with the VR game application, you can record an MR video as if the character entered the world of the game application by combining the game application screen with the VirtualMotionCapture screen.
This time, I explain how to become Vtuber by VirtualMotionCapture and the STYLY scene.

The completion image

Introduction and preparation

Introduction

In order to record the MR movie that combines the screens of the VR-game application and VirtualMotionCapture, you need to use the application for MR overlay such as LIV, and the MR-overlay function by using externalcamera.cfg.
I will explain how to use those applications and how to set them in the next article, so I explain the basics to use VirtualMotionCapture in this article.

Preparation

VirtualMotionCapture is the application that handles the 3D model of the file format called ‘VRM’, so you need to prepare a 3D model in VRM format.
In addition to the models sold and distributed, you can also use your original VRM model created with the tools such as Vroid, Cecil Transform Application.

▼This article explains how to use VRoid▼

And, we use a live-streaming application called OBS Studio at the final stage, so download this software as well.
However, if you use other live-streaming software, you can use it instead.

▼Download OBS Studio from here▼
https://obsproject.com/ja/download

What is VirtualMotionCapture?

It is an application that allows you control 3D model of VRM by VR device (HTC Vive/Oculus Rift/WinMR) so you can become so-called Vtuber.
Unlike other common applications that control the model by VR devices, it can control the model while playing a VR game.

Reference: https://sh-akira.booth.pm/items/999760

The most notable feature of VirtualMotionCapture is that you can launch it simultaneously with the VR-game application, which is considered impossible with regular applications.
By using this feature, you can move a character in sync with the motion of the player who plays the VR game, so you can livestream the VR game like a Vtuber.
In addition, instead of launching it with the VT game, you can simply use it to control a 3D model, which is also an advantage for various applications.

*Note by the translator – VirtualMotionCapture supports English. You can select the language by the drop-down box at the bottom of  the ‘Settings’ window.

Download VirtualMotionCapture

You can download VirtualMotionCapture for free from BOOTH.
And, if you are a paid subscription member, the pre-release version can be downloaded from pixv FUNBOX.
This time, I download the free version (Ver 0.23) from BOOTH.

▼BOOTH▼
https://sh-akira.booth.pm/items/999760

Import VRM model

After launching VirtualMotionCapture, firstly, import the VRM model you want to control.
When launching the application, it also brings up the control panel at the same time.

Control Panel

Press the ‘Open VRM’ button on the ‘Settings’ tab to open the window where you select the file.

Select the VRM file by pressing the Open VRM’ button

Select and open the VRM file prepared earlier.
It brings up the license information for the avatar, so check it closely before pressing the ‘Agree and Download’ button if you use the purchased model etc.

The license information is displayed

In the case you use your original model created with VRoid etc., the avatar information you set by yourself is displayed, so just press the ‘Agree and Download’ button.
If everything goes well, the model will be displayed on the screen.

The model has been imported

The setting for Camera, Lip sync and Trackers

The setting for Camera

You can set up the Camera by selecting the ‘Camera’ tab.
Press ‘Front’ as we want use the Front Camera.
Click the Scene screen and then use the mouse to adjust the position of the camera.

The setting for the Camera

The setting for Lip Sync

By setting up ‘Lip Sync’, you can animate the lip of the avatar in sync with the voice input by the microphone.
If you have not specified the microphone for Lip Sync, the ‘Lip Sync’ tab is shown in red, so you can easily see whether it’s set up or not.

The ‘Lip Sync’ tab – The microphone has not been specified.

 

The ‘Lip Sync’ tab – The microphone has been specified

If you don’t use Lip Sync, you don’t need to set it up.

The setting for Trackers

VirtualMotionCapture also supports ‘full tracking’ by Trackers.
In order to assign the Trackers, press the Setting button on the Settings tab at first.
Next, press the ‘Tracker Config’ button to open the ‘Tracker assignment setting’ window where you can assign the Trackers.
As we don’t use Trackers this time, leave the default settings.

The ‘Setting’ window

 

The ‘Tracker assignment setting’ window

Calibration

After finishing the settings, let’s move the model straight away.
Press the ‘Calibration’ button on the ‘Settings’ tab.

The settings for Calibration

There are three modes for the calibration. This time, select the ‘Normal’ mode.
The calibration starts after 5 seconds when pressing the ‘Calibration’ button or pulling the trigger of the controller.
Spread your arms, keep the controllers upright and turn your palms forwards.
Although the screen instructs ‘take the same pose as the model displayed…’, the angle of the wrists would go wrong if you turn the controller and your palms downward.
The white spheres displayed in the main screen represents the locations of the HMD and the controllers.
Adjust the position of the HMD and the controllers so that those spheres are at the head and hands of the model.

The spheres show the locations of the HMD and the controllers

You will be able to control the model if the calibration is successful.

The calibration is successful

How to become Vtuber by using the space created with STYLY

Now you came to be able to move the model.
Next, create the live-streaming screen for Vtuber by using the space created with STYLY as the background.

The setting for background colour

By selecting the ‘Background’ tab, you can change the background colour to ‘blueback’, ‘greenback’ or other colours.
If you select one of those background colours, you can combine the screen of the character controlled by VirtualMotionCapture with the image or the movies you want to use as the background by using the ‘Chroma key’ processing.
In addition, you can select ‘transparent’ as the background of VirtualMotionCappture, so it’s possible to overlay the character on the desktop screen.
This time, we apply ‘Chromakey’ to the screen of VirtualMotionCapture by using ‘greenback’ to combine the character with the space created with STYLY.

Select the background colour

Prepare a space by using STYLY

Create your favourite space with STYLY Editor.
As we use the space as the background, you can create a dummy space by modelling what the camera can see only.
As you can record the movie of the space by using the Camera function in STYLY Editor, you can make the background with motions.
This time, we use what is shown in the screenshot below.

Combine Character and Space

Finally, we overlay the character on the screenshot of the scene created by STYLY Editor.
We use the live-streaming software called OBS Studio this time. Launch VirtualMotionCapture before launching OBS.
Firstly, add the image to ‘Sources’.
Press the ‘+’ button below the ‘Sources’ box. Select ‘Image’ from the menu and select  ‘Create new’.

Add the image

 

Create a new ‘Image’

Then, you can add the background image to ‘Sources’ by selecting the image you want to use in the ‘Properties’ window.

Select the image

 

Now the image has been added

Next, add the window of VirtualMotionCapture to ‘Sources’.
As we did earlier, press the ‘+’ button below the ‘Sources’ box to display the menu.
In this time, select ‘Window Capture’ from the menu and select ‘Create new’.

Select ‘Window Capture’

 

Create a new ‘Window Capture’

Then, select ‘[VirtualMotionCapture.exe]: VirtualMotionCapture v0.23’ in the Property window.
It adds the window of VirtualMotionCapture to ‘Sources’.

Select the window of VirtualMotionCapture

As a Source listed above another Source will be on the top of another Source in the display, reorder the Sources so that the Window Capture is above the Image.
In this state, only the screen of VirtualMotionCapture is visible and you cannot see the background image at all, so we apply ‘Chroma key’ to the VirtualMotionCapture window to show the background.

How it looks – before applying ‘Chroma Key’

Right-click the Window Capture in ‘Sources’ and select ‘Filters’ from the menu.
Next, Press the ‘+’ button below the ‘Effect Filters’ box and select ‘Chroma Key’ from the menu.

Select ‘Chroma Key’

Selecting the ‘Chroma Key’ in the Effect Filter box and set the same background colour as the selected in VirtualMotionCapture for “Key Color Type’.
Then, the ‘Chroma Key’ filter makes the background colour of VirtualMotionCapture transparent so that the character is combined with the background and displayed.

‘Chroma Key’ filter

 

How it looks after applying ‘Chroma Key’

It’s completed now. Once you set up the recording and the broadcasting, you will be able to create a movie and livestream.

Completed!

Congratulations!
In this article, I introduced the basics to use VirtualMotionCapture.
Even this basic method allows you to livestream a gameplay like Vtuber straight away.
However, you still need to do more if you want to create a MR video as if the character entered the world of a game application.
Next, I will introduce the application required to record an MR video and advanced use of VirtualMotionCapture.