How to synchronize sound and shader by using Unity’s Timeline


I explain how to synchronize sound with a shader by using ‘Timeline’ in Unity.


A representation of a noise-style shader synchronized with noise-style music is shown below.




Github Sample Project

Play sound with Timeline

Here I explain how to create a Timeline and play sound.

Create Timeline

Firstly, create a Timeline. Right-click on the Project window and select ‘Create > Timeline’.


Create Timeline


The Timeline asset created

Convert Timeline asset into an object

Next, convert the Timeline asset into an object. Drag the Timeline asset created earlier into the Hierarchy window.


Drag the Timeline asset into the Hierarchy window

 It creates a GameObject with PlayableDirector. You need PlayableDirector to enable Timeline.


The created GameObject with PlayableDirectory

Double-click the Timeline asset to open the Timeline window.


Double-click the Timeline asset

In the Timeline window, you can edit and play the Timeline. You basically work on this window when editing the Timeline.


Timeline window


Play sound

In this time, I use the sound that plays the sine wave at regular intervals.

You can download the sample sound (Link). Firstly, drag the sound data (AudioClip) into the Timeline.

 It creates ‘AudioTrack’ and assigns ‘AudioClip’ to the Timeline.


AudioClip is assigned to Timeline

Drag the ‘sin_C5’ clip so that its left end is at 0.


Adjust the starting point of the Audio Clip to 0

In this state, you can play the sound by pressing the Play button in Unity.

Lock Timeline

To avoid unselecting the Timeline when selecting other objects, lock the Timeline by clicking the ‘Padlock’ icon in the top-right corner of the window.


Lock Timeline

Create Shader

Next, create a shader that is controlled by the Timeline. Right-click the Project window to open the context menu and select ‘Create > Shader > Unlit Shader’ to create a shader.


Create Shader

Name it ‘SimpleColor’.



  Edit the shader as shown below.

Shader "Unlit/SimpleColor"
		_MainTex("Texture", 2D) = "white" {}
		_Color("Color", Color) = (0, 0, 0, 1) // color
			Tags { "RenderType" = "Opaque" }
			LOD 100

				#pragma vertex vert
				#pragma fragment frag
				// make fog work
				#pragma multi_compile_fog

				#include "UnityCG.cginc"

				struct appdata
					float4 vertex : POSITION;
					float2 uv : TEXCOORD0;

				struct v2f
					float2 uv : TEXCOORD0;
					float4 vertex : SV_POSITION;

				sampler2D _MainTex;
				float4 _MainTex_ST;
				float4 _Color; // color

				v2f vert(appdata v)
					v2f o;
					o.vertex = UnityObjectToClipPos(v.vertex);
					o.uv = TRANSFORM_TEX(v.uv, _MainTex);
					return o;

				fixed4 frag(v2f i) : SV_Target
					return tex2D(_MainTex, i.uv) * _Color;


Create Material

Create a material from the above SimpleColor.


Create Material


The created material


Create a Cube

Press the Create button in the Hierarchy window and select ‘3D Object > Cube’ to create a Cube.


Create a Cube


Attach Material to Cube

Attach the material to the Cube by dragging the material into it.


Attach the material to the Cube

When attaching the material, the Cube turns black.


The result – The material has been attached


Change Shader with Timeline

Assign Cube to Timeline

Drag the Cube into the Timeline to assign the Cube to the Timeline.


Drag Cube into Timeline

 It brings up the menu. Select ‘Animation Track’.

Animation Track を選択

Select ‘Animation Track’

It creates the AnimationTrack for the Cube object in the Timeline.


The created AnimationTrack

That’s all you need to assign the Cube object to the Timeline. Next, we are going to edit this ‘AnimationTrack’. Set the position of the ‘Seek Bar'(=the Playhead) to 0.


Set the position of the ‘Seek Bar’ to 0.

Alternatively, you can set the ‘Seek Bar’ position to zero by pressing the ‘<<‘ button on the Timeline.


The button to set the Playhead position to zero


Add a clip to Timeline

Set Timeline to the ‘Recording’ mode

Click the red button in the upper left corner of the Timeline window to set the Timeline to the Recording mode.


Start Recording

Then, the AnimationTrack turns red, which means it became the Recording mode and it’s editable now.


The Timeline in the ‘Recording’ mode


Add a ‘Black’ keyframe

Next, select the Cube.


Click the Cube to select

Right-click ‘Color’ in the Material field of the Inspector window.


Right-click ‘Color’ in the Inspector window

 It brings up the menu. Select ‘Add Key’.

Add Keyを選択

Select ‘Add Key’

It added the current Color information to the Timeline as a keyframe.


A keyframe is added at the position of the Timeline ‘Seek Bar’.

In this time, the ‘Black’ keyframe, ‘Color = (0, 0, 0, 1)’, has been added to the Timeline.  

Convert into ‘Clip Track’

We added a keyframe just now. Then, let’s convert it into ‘Clip Track’ to make it easy to handle.


Right-click the Animation Track

In the displayed menu, select ‘Convert to Clip Track’.


Convert into ‘ClipTrack’

It changes the keyframe into ‘Clip Track’.

作成されたClip Track

The created Clip Track

Double-click the Clip Track to open the Animation window and the Clip Track. This time, The ‘Color=(0, 0, 0, 1)’ keyframe is placed at 0:00.


Duplicate Clip Track

Right-click the Clip Track and select ‘Duplicate’ from the menu displayed.


Duplicate ClipTrack



The duplicated ClipTrack


Rename the Clip

The name of the clip is not so descriptive at the moment, so we rename it. Firstly, click the first Recorded clip.


Click the first Recorded clip

Enter ‘Black’ in the Name field (It’s been set to ‘Recorded’ by default) at the top of the Inspector window.


Change ‘Recorded’ into ‘Black’

Once you renamed, the text on the clip turns into ‘Black’.


The text on the clip turned into ‘Black’

Similarly, rename the second clip ‘White’.


Result – the clips have been renamed


Make Keyframe white

Double-click the White clip to open.


Double-click the White clip to open

Set the keyframe shown in the window to ‘Color = (1, 1, 1, 1)’.

キーフレームをColor = (1, 1, 1, 1)にする

Set the keyframe to ‘Color = (1, 1, 1, 1)’

If you play the Timeline in this state, the cube turns white ‘Color = (1,1,1,1)’ when the White track starts.


Cube turns white when the White track starts

Now the colour of the Cube changes but it’s not synchronized with the timing of the playing sound.


Synchronize the White track with sound

We synchronize the sound and the colour change by synchronizing the waveform and the White track. Since the waveform is displayed too small to adjust, scale it up by using the mouse wheel.


Use the mouse wheel to scale it up

Scale it up so that you can see the waveform.


Scale it up until you can see the waveform

Next, we adjust the White clip to the timing when the sound starts, which is indicated by the red frame below.

Because ‘Snap to Frame’ is enabled by default, the track moves using frames when matching the track to the sound. 


When moving the track, the track is snapped to a frame.

You need to turn off ‘Snap to Frame’ to solve it.


Disable ‘Snap’

Click the Cog icon in the top right corner of the window.


Click the Cog icon

You can find ‘Snap to Frame’ in the menu displayed, so uncheck it.

Snap to Frameのチェックを外す

Uncheck ‘Snap to Frame’

You can move the track to anywhere as you disabled ‘Snap to Frame’.


Move the track with ‘Snap to Frame’ disabled


Adjust the White track to the timing when the sound starts

As shown below, place the White track so that it’s synchronized with the sound.

In this state, when playing back the scene, the cube turns white as the sound starts.

Anyway, let’s save the scene here. You can save the scene by selecting ‘File > Save Scene’ from the menu in the top left corner of the window.


Save the scene


Convert Animation clip into Asset

If you converted the clip created in the Timeline into an asset, it would be more convenient as it could be shared for other use.  

Name Asset

Firstly, select the Black clip and enter ‘SetBlack’ for ‘Animation Clip Name’ in the Inspector. Similarly, select the White clip and enter ‘SetWhite’ for ‘Animation Clip Name’ in the Inspector.


Enter ‘SetBlack’ in the ‘Animation Clip Name’ field

In this state, ‘AnimationClip’ is created in the Timeline asset by saving the scene.


‘AnimationClip’ appeared in the Timeline


Troubleshooting: The Name field of AnimationClip doesn’t show up

If you didn’t save the scene, the Name field of AnimationClip wouldn’t appear.


The Name field of AnimationClip wouldn’t appear if you didn’t save the scene.

You need to save the scene to solve it.


Save the scene


Convert AnimationClip to Asset

Select and duplicate the AnimationClip in the Timeline to take it out from the Timeline and convert it into an asset.


Duplicate AnimationClip

Then, the SetBlack animation is converted into an asset and the ‘SetBlack.anim’ file is created.


The AnimationClip converted into an asset

You also create ‘SetWhite.anim’ in the same way.  

Place AnimationClip in Timeline

Drag the created SetWhite.anim and SetBlack.anim into the Timeline.


Assign ‘SetBlack.anim’ to ‘Timeline’

It creates the clip in the Timeline, but it looks too short.


The SetBlack clip created in the Timeline


Copy the length of other animation track

As the length of the Black track seems to be relevant, I want to adjust the SetBlack tract to the same length. Firstly, select the Black clip and copy the ‘Duration’ value in the Inspector.


Copy the ‘Duration’ value

Select the SetBlack clip and paste it into the ‘Duration’ field.


Paste the value into ‘Duration’

Now the length of the SetBlack clip is the same as the Black clip.


How the SetBlack clip looks after copying the Duration value

Similarly, place SetBlack clips and SetWhite clips alternately with no gaps.


SetBlack and SetWhite placed alternately

In this state, when playing back the scene, the colour turns white and black alternately, synchronizing with the sound.

That’s all you need to synchronize the shader with the sound. 

How to upload the asset from Unity to STYLY (For Scenes)

By uploading the Scene with the Timeline object placed, you can upload the Timeline to STYLY. Please refer to the URL below to learn more about how to upload an asset.

How to upload the asset from Unity to STYLY (For Prefab)

If you want to upload the Timeline object as a Prefab, you need to put what the Timeline object refers to, into the Prefab. The Timeline object refers to the Cube this time, so put the Cube into the Timeline object.


Put the Cube into the Timeline object

Please refer to the URL below to learn more about how to upload an asset. 

The setting for the sound to upload it to STYLY

If you uploaded a large sound data to STYLY, the changes of the shader and the sound would sometimes get out of sync because of the latency when playing back the sound. You can solve this latency by using the setting shown below; Use the ‘.wav‘ sound file; Check ‘Load in background‘ (i.e. load the sound with the other thread); Set ‘Compression Format‘ to PCM.