How to synchronize sound and shader by using Unity’s Timeline

Introduction

Example

A representation of a noise-style shader synchronized with noise-style music is shown below.

Sample

 

 

Github Sample Project
https://github.com/styly-dev/STYLY-Unity-Examples/tree/master/Assets/STYLY_Examples/NoiseMusicSync_Sample

Convert Timeline asset into an object

Next, convert the Timeline asset into an object. Drag the Timeline asset created earlier into the Hierarchy window.

 It creates a GameObject with PlayableDirector. You need PlayableDirector to enable Timeline.

Double-click the Timeline asset to open the Timeline window.

In the Timeline window, you can edit and play the Timeline. You basically work on this window when editing the Timeline.

 

Play sound

In this time, I use the sound that plays the sine wave at regular intervals.

You can download the sample sound (Link). Firstly, drag the sound data (AudioClip) into the Timeline.

 It creates ‘AudioTrack’ and assigns ‘AudioClip’ to the Timeline.

Drag the ‘sin_C5’ clip so that its left end is at 0.

In this state, you can play the sound by pressing the Play button in Unity.

Lock Timeline

To avoid unselecting the Timeline when selecting other objects, lock the Timeline by clicking the ‘Padlock’ icon in the top-right corner of the window.

Create Shader

Next, create a shader that is controlled by the Timeline. Right-click the Project window to open the context menu and select ‘Create > Shader > Unlit Shader’ to create a shader.

Name it ‘SimpleColor’.

  Edit the shader as shown below.

Shader "Unlit/SimpleColor"
{
	Properties
	{
		_MainTex("Texture", 2D) = "white" {}
		_Color("Color", Color) = (0, 0, 0, 1) // color
	}
		SubShader
		{
			Tags { "RenderType" = "Opaque" }
			LOD 100

			Pass
			{
				CGPROGRAM
				#pragma vertex vert
				#pragma fragment frag
				// make fog work
				#pragma multi_compile_fog

				#include "UnityCG.cginc"

				struct appdata
				{
					float4 vertex : POSITION;
					float2 uv : TEXCOORD0;
				};

				struct v2f
				{
					float2 uv : TEXCOORD0;
					UNITY_FOG_COORDS(1)
					float4 vertex : SV_POSITION;
				};

				sampler2D _MainTex;
				float4 _MainTex_ST;
				float4 _Color; // color

				v2f vert(appdata v)
				{
					v2f o;
					o.vertex = UnityObjectToClipPos(v.vertex);
					o.uv = TRANSFORM_TEX(v.uv, _MainTex);
					UNITY_TRANSFER_FOG(o,o.vertex);
					return o;
				}

				fixed4 frag(v2f i) : SV_Target
				{
					return tex2D(_MainTex, i.uv) * _Color;
				}
				ENDCG
			}
		}
}

 

Create Material

Create a material from the above SimpleColor.

Create a Cube

Press the Create button in the Hierarchy window and select ‘3D Object > Cube’ to create a Cube.

 

Attach Material to Cube

Attach the material to the Cube by dragging the material into it.

When attaching the material, the Cube turns black.

 

Change Shader with Timeline

Assign Cube to Timeline

Drag the Cube into the Timeline to assign the Cube to the Timeline.

 It brings up the menu. Select ‘Animation Track’.

It creates the AnimationTrack for the Cube object in the Timeline.

That’s all you need to assign the Cube object to the Timeline. Next, we are going to edit this ‘AnimationTrack’. Set the position of the ‘Seek Bar'(=the Playhead) to 0.

Alternatively, you can set the ‘Seek Bar’ position to zero by pressing the ‘<<‘ button on the Timeline.

 

Add a clip to Timeline

Set Timeline to the ‘Recording’ mode

Click the red button in the upper left corner of the Timeline window to set the Timeline to the Recording mode.

Then, the AnimationTrack turns red, which means it became the Recording mode and it’s editable now.

 

Add a ‘Black’ keyframe

Next, select the Cube.

Right-click ‘Color’ in the Material field of the Inspector window.

 It brings up the menu. Select ‘Add Key’.

It added the current Color information to the Timeline as a keyframe.

In this time, the ‘Black’ keyframe, ‘Color = (0, 0, 0, 1)’, has been added to the Timeline.  

Convert into ‘Clip Track’

We added a keyframe just now. Then, let’s convert it into ‘Clip Track’ to make it easy to handle.

In the displayed menu, select ‘Convert to Clip Track’.

It changes the keyframe into ‘Clip Track’.

Double-click the Clip Track to open the Animation window and the Clip Track. This time, The ‘Color=(0, 0, 0, 1)’ keyframe is placed at 0:00.

 

Duplicate Clip Track

Right-click the Clip Track and select ‘Duplicate’ from the menu displayed.

 

 

Rename the Clip

The name of the clip is not so descriptive at the moment, so we rename it. Firstly, click the first Recorded clip.

Enter ‘Black’ in the Name field (It’s been set to ‘Recorded’ by default) at the top of the Inspector window.

Once you renamed, the text on the clip turns into ‘Black’.

Similarly, rename the second clip ‘White’.

 

Make Keyframe white

Double-click the White clip to open.

Set the keyframe shown in the window to ‘Color = (1, 1, 1, 1)’.

If you play the Timeline in this state, the cube turns white ‘Color = (1,1,1,1)’ when the White track starts.

Now the colour of the Cube changes but it’s not synchronized with the timing of the playing sound.

 

Synchronize the White track with sound

We synchronize the sound and the colour change by synchronizing the waveform and the White track. Since the waveform is displayed too small to adjust, scale it up by using the mouse wheel.

Scale it up so that you can see the waveform.

Next, we adjust the White clip to the timing when the sound starts, which is indicated by the red frame below.

Because ‘Snap to Frame’ is enabled by default, the track moves using frames when matching the track to the sound. 

You need to turn off ‘Snap to Frame’ to solve it.

 

Disable ‘Snap’

Click the Cog icon in the top right corner of the window.

You can find ‘Snap to Frame’ in the menu displayed, so uncheck it.

You can move the track to anywhere as you disabled ‘Snap to Frame’.

 

Adjust the White track to the timing when the sound starts

As shown below, place the White track so that it’s synchronized with the sound.

In this state, when playing back the scene, the cube turns white as the sound starts.

Anyway, let’s save the scene here. You can save the scene by selecting ‘File > Save Scene’ from the menu in the top left corner of the window.

 

Convert Animation clip into Asset

If you converted the clip created in the Timeline into an asset, it would be more convenient as it could be shared for other use.  

Name Asset

Firstly, select the Black clip and enter ‘SetBlack’ for ‘Animation Clip Name’ in the Inspector. Similarly, select the White clip and enter ‘SetWhite’ for ‘Animation Clip Name’ in the Inspector.

In this state, ‘AnimationClip’ is created in the Timeline asset by saving the scene.

 

Troubleshooting: The Name field of AnimationClip doesn’t show up

If you didn’t save the scene, the Name field of AnimationClip wouldn’t appear.

You need to save the scene to solve it.

 

Convert AnimationClip to Asset

Select and duplicate the AnimationClip in the Timeline to take it out from the Timeline and convert it into an asset.

Then, the SetBlack animation is converted into an asset and the ‘SetBlack.anim’ file is created.

You also create ‘SetWhite.anim’ in the same way.  

Place AnimationClip in Timeline

Drag the created SetWhite.anim and SetBlack.anim into the Timeline.

It creates the clip in the Timeline, but it looks too short.

 

Copy the length of other animation track

As the length of the Black track seems to be relevant, I want to adjust the SetBlack tract to the same length. Firstly, select the Black clip and copy the ‘Duration’ value in the Inspector.

Select the SetBlack clip and paste it into the ‘Duration’ field.

Now the length of the SetBlack clip is the same as the Black clip.

Similarly, place SetBlack clips and SetWhite clips alternately with no gaps.

In this state, when playing back the scene, the colour turns white and black alternately, synchronizing with the sound.

That’s all you need to synchronize the shader with the sound. 

How to upload the asset from Unity to STYLY (For Scenes)

By uploading the Scene with the Timeline object placed, you can upload the Timeline to STYLY. Please refer to the URL below to learn more about how to upload an asset.

How to upload the asset from Unity to STYLY (For Prefab)

If you want to upload the Timeline object as a Prefab, you need to put what the Timeline object refers to, into the Prefab. The Timeline object refers to the Cube this time, so put the Cube into the Timeline object.

Please refer to the URL below to learn more about how to upload an asset. 

The setting for the sound to upload it to STYLY

If you uploaded a large sound data to STYLY, the changes of the shader and the sound would sometimes get out of sync because of the latency when playing back the sound. You can solve this latency by using the setting shown below; Use the ‘.wav‘ sound file; Check ‘Load in background‘ (i.e. load the sound with the other thread); Set ‘Compression Format‘ to PCM.