How to become a Vtuber with ‘Virtual Motion Capture’

In this article, I introduce an application called VirtualMotionCapture.
VirtualMotionCapture is an application to control a 3D model in VRM format by VR devices, and I recommend it especially for those who want to become ‘Vtuber’.
In addition, since it can be launched simultaneously with the VR game application, you can record an MR video as if the character entered the world of the game application by combining the game application screen with the VirtualMotionCapture screen.
This time, I explain how to become a Vtuber with VirtualMotionCapture and the space generated by STYLY.

The completion image

Introduction and preparation

Introduction

In order to record the MR video that combines the screens of the VR-game application and VirtualMotionCapture, you need to use the application for MR-overlay such as LIV, and the MR-overlay function by using externalcamera.cfg.
I will explain how to use those applications and how to set up them in the next article, so I introduce the basics to use VirtualMotionCapture in this article.

Preparation

VirtualMotionCapture is the application that handles the 3D model defined by the file format called ‘VRM’, so you need to prepare a 3D model in VRM format.
In addition to the models sold or distributed, you can also use your original VRM model created with the tools such as Vroid, ‘Cecil Henshin Application’.

▼This article explains how to use VRoid▼

And, we use a live-streaming application called OBS Studio at the final stage, so download this software as well.
However, if you use other live-streaming software, you may use it instead.

▼Download OBS Studio from here▼
https://obsproject.com/ja/download

What is VirtualMotionCapture?

It is an application that allows you control VRM-format 3D model by VR device (HTC Vive/Oculus Rift/WinMR), so you can become so-called Vtuber.
Unlike other common applications that control the model by VR devices, it can control the model while playing a VR game.

Reference: https://sh-akira.booth.pm/items/999760

The most notable feature of VirtualMotionCapture is that you can launch it simultaneously with the VR-game application, which is considered impossible with other common applications.
By using this feature, you can move a character in sync with the motion of the player who plays the VR game, so you can livestream the VR game like a Vtuber.
In addition, instead of launching it with the VR game, you can simply use it to control a 3D model, which is also an advantage for various applications.

*Note by the translator – VirtualMotionCapture supports English UI. You can select the language by the drop-down box at the bottom of the ‘Settings’ window.

Download VirtualMotionCapture

You can download VirtualMotionCapture for free from BOOTH.
And, if you are a paid subscription member, the pre-release version can be downloaded from pixv FUNBOX.
This time, I download the free version (Ver 0.23) from BOOTH.

▼BOOTH▼
https://sh-akira.booth.pm/items/999760

Import VRM model

After launching VirtualMotionCapture, import the VRM model you want to control.
When launching the application, it also brings up the control panel at the same time.

Control Panel

Press the ‘Open VRM’ button on the ‘Settings’ tab to open the ‘file-selection’ window.

Select the VRM file by pressing the ‘Open VRM’ button

Select and open the VRM file prepared earlier.
It brings up the license information for the avatar, so check it closely before pressing the ‘Agree / Import’ button if you use the purchased model etc.

The license information is displayed

In the case you use your original model created with VRoid etc., the avatar information you set is displayed, so just press the ‘Agree / Import’ button.
If everything goes well, the model will be displayed on the screen.

The model has been imported

The setting for Camera, Lip Sync and Trackers

The setting for Camera

You can set up the Camera by selecting the ‘Camera’ tab.
Press ‘Front’ as we want to use the Front Camera.
Click the main screen and then use the mouse to adjust the position of the Camera.

The setting for the Camera

The setting for Lip Sync

By setting up ‘Lip Sync’, you can animate the lip of the avatar in sync with the voice input by the microphone.
If you have not specified the microphone for Lip Sync, the ‘Lip Sync’ tab is shown in red, so you can easily see whether it’s set up or not.

The ‘Lip Sync’ tab – The microphone has not been specified.

 

The ‘Lip Sync’ tab – The microphone has been specified

If you don’t use Lip Sync, you don’t need to set it up.

The setting for Trackers

VirtualMotionCapture also supports ‘full tracking’ by Trackers.
In order to assign the Trackers, press the ‘Setting’ button on the ‘Settings’ tab.
Next, press the ‘Tracker Config’ button to open the ‘Tracker assignment setting’ window where you can assign the Trackers.
As we don’t use Trackers this time, leave it with the default settings.

The ‘Setting’ window

 

The ‘Tracker assignment setting’ window

Calibration

After finishing the settings, let’s move the model straight away.
Press the ‘Calibration’ button on the ‘Settings’ tab.

The settings for Calibration

There are three modes for the calibration. This time, select the ‘Normal’ mode at the top.
The calibration starts after 5 seconds when pressing the ‘Calibration’ button or pulling the trigger of the controller.
Spread your arms, keep the controllers upright and turn your palms forwards.
Although the screen instructs ‘take the same pose as the model displayed…’, the angle of the wrists would go wrong if you turn the controller and your palms downward.
The white spheres displayed in the main screen represents the positions of the HMD and the controllers.
Adjust the position of the HMD and the controllers so that those spheres are at the head and hands of the model.

The spheres represent the positions of the HMD and the controllers

You can control the model if the calibration is successful.

The calibration is successful

How to become Vtuber with STYLY’s space

Now you came to be able to move the model.
Next, create the live-streaming screen for Vtuber by using the space created with STYLY as the background.

The setting for background colour

By selecting the ‘Background’ tab, you can change the background colour to ‘blueback’, ‘greenback’ or other colours.
If you select one of those background colours, you can combine the character controlled by VirtualMotionCapture with the image or the video you want to use as the background, by using the ‘Chroma key’ processing.
In addition, you can select ‘transparent’ as the background of VirtualMotionCapture, so it’s possible to overlay the character on the desktop screen.
This time, by using ‘greenback’, we apply ‘Chroma key’ to the screen of VirtualMotionCapture to combine the character with the space created with STYLY.

Select the background colour

Prepare a space with STYLY

Create your favourite space with STYLY Editor.
Since we use the space as the background, you may create a dummy space by modelling only what the camera can see.
As you can record the video of the space with the Camera function in STYLY Editor, you can also make the background with motions.
This time, we use what is shown in the screenshot below.

Combine Character and Space

Finally, we overlay the character on the screenshot of the scene created with STYLY Editor.
We use the live-streaming software called OBS Studio this time. Launch VirtualMotionCapture before launching OBS Studio.
Firstly, we add the image to ‘Sources’.
Press the ‘+’ button below the ‘Sources’ box. Click ‘Image’ from the menu and select  ‘Create new’.

Add the image

 

Create a new ‘Image’

Then, you can add the background image to ‘Sources’ by selecting the image you want to use in the ‘Properties’ window.

Select the image

 

Now the image has been added

Next, add the window of VirtualMotionCapture to ‘Sources’.
As we did earlier, press the ‘+’ button below the ‘Sources’ box to display the menu.
In this time, Click ‘Window Capture’ from the menu and select ‘Create new’.

Select ‘Window Capture’

 

Create a new ‘Window Capture’

Then, select ‘[VirtualMotionCapture.exe]: VirtualMotionCapture v0.23’ in the Property window.
It adds the window of VirtualMotionCapture to ‘Sources’.

Select the window of VirtualMotionCapture

As a Source listed above another Source will be on the top of another Source in the display, reorder the Sources so that the Window Capture is above the Image.
In this state, only the screen of VirtualMotionCapture is visible and you cannot see the background image at all, so we apply ‘Chroma key’ to the VirtualMotionCapture window to show the background.

How it looks – before applying ‘Chroma Key’

Right-click the Window Capture in ‘Sources’ and select ‘Filters’ from the menu.
Next, press the ‘+’ button below the ‘Effect Filters’ box and select ‘Chroma Key’ from the menu.

Select ‘Chroma Key’

Select the ‘Chroma Key’ in the ‘Effect Filter’ box, and set ‘Key Color Type’ to the same colour as the background selected in VirtualMotionCapture.
Then, the ‘Chroma Key’ filter makes the background colour of VirtualMotionCapture transparent so that the character is combined with the background image in the display.

‘Chroma Key’ filter

 

How it looks after applying ‘Chroma Key’

It’s completed now. Once you set up the recording and the broadcasting, you will be able to create a video and livestream.

Completed!

Congratulations!
In this article, I introduced the basics to use VirtualMotionCapture.
Even this basic way allows you to livestream gameplay like Vtuber straight away.
However, you still need to learn more if you want to create an MR video as if the character entered the world of the game application.
Next, I will introduce the application required to record an MR video and advanced way to use VirtualMotionCapture.