-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathindex.html
More file actions
104 lines (100 loc) · 6.4 KB
/
index.html
File metadata and controls
104 lines (100 loc) · 6.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
<html>
<head>
<title>Music Visual</title>
<script src="pixi.js"></script>
</head>
<body>
<link rel="stylesheet" type="text/css" href="musicvis.css"></link>
<audio id="music-player"></audio>
<div class="music-container">
<div class="music-center">
<h2>Neural Style Music Visualizer</h2>
<div id="music-visualizer-div"></div>
<div class="music-input">
<button id="music-playpause">Play</button>
</div>
</div>
<h3>Controls</h3>
<div class="music-left">
<div class="music-input">
Background:
<select id="music-selectbg">
<option value="0">Galaxy (Picasso / Kandinsky)</option>
<option value="1">Water (Van Gogh / Munch)</option>
<option value="2">Debug Blob (no style)</option>
</select>
</div>
<div class="music-input">
Song:
<select id="music-selectsong">
<option value="0">Ready err Not (Flying Lotus)</option>
<option value="1">Introductory Nomenclature (Telefon Tel Aviv)</option>
<option value="2">Debug Sinestep (no artist)</option>
</select>
</div>
<div class="music-input">
Cut Frequency between background styles:
<input id="music-selectfreq" type="range" min="100" max="3000" step="50" value="550"></input>
<div id="music-freqdisplay">550 Hz</div>
</div>
<div class="music-input">
Blending of different styles (slide right for more blending):
<input id="music-selectblend" type="range" min="0.1" max="0.5" step="0.05" value="0.3"></input>
</div>
<div class="music-input">
Smoothing when transitioning styles (slide right for quicker transitions):
<input id="music-selectsmooth" type="range" min="0.001" max="0.006" step="0.0005" value="0.004"></input>
</div>
<div class="music-input">
Lowpass particle threshold (slide right for fewer blue particles):
<input id="music-selectlowpart" type="range" min="1000" max="7000" step="100" value="4000"></input>
</div>
<div class="music-input">
Highpass particle threshold (slide right for fewer red particles):
<input id="music-selecthighpart" type="range" min="1000" max="7000" step="100" value="1500"></input>
</div>
<h3>Information</h3>
<div>
<p><strong>Details</strong>: This project uses <a href="https://github.com/jcjohnson/neural-style">an implementation of the
Neural Style paper</a> in conjunction with Python signal processing libraries and <a href="http://www.pixijs.com/">Pixi.js</a>
to create a music visualizer. When a song's bass tones are dominating, the background will show one style of some ambient
background, and when the treble tones are dominating, the background will show a different style for the same background.
"In-between" tones will present a blend of the two styles. Additionally, particles will appear for strong bass (blue) and
treble (red) noises. These particles are initialized with a random speed and direction but they follow a vector field formed
by the gradient of the background image. </p>
<p><strong>Notes</strong>: This is not a real-time system, so if you want to add more backgrounds or music, you will have to generate images or
data files using Python scripts.</p>
<ul>
<li>To create a set of 5 background images using 2 different styles, first clone and install the
<a href="https://github.com/jcjohnson/neural-style">neural-style</a> repo, copy <em>gen_bg.py</em> to that directory, modify
the paths in that script to point to the desired content/style images, and then run it.</li>
<li>To create a vector field file for the particle systems, run <em>image_gradient.py</em> after modifying that script (bottom)
to point to the right image file.</li>
<li>To create a frequency data file for a song, run <em>filter_music.py</em> after modifying that script (bottom) to point
to the right music file (note that the music needs to be in WAV format). The frequency cutoffs that decide when bass or treble
particle systems are created are also selected at this stage (in comparison, the background style cutoff is decided by the client). </li>
</ul>
<p>If you create new content, you will need to modify the global variables at the top of <em>musicvis.js</em> so that you will be able
to select them on this page. To run this system, you need to have a server with PHP serving this directory. XAMPP is a good option
for this task. If PHP is not available, a separate JS file (<em>readclient.js</em>) can be used to bypass the need for <em>webservice.php</em>.
The sliders should work as the song is playing; however, to choose a new song or background, wait until the current song is over or
refresh the page. If the renderer blacks out after multiple plays, or the audio and video are desynced, please refresh the page (there are
a bunch of small problems that prevent the system from being robust; this prototype is intended to show the general idea of the project).
</p>
<p><strong>Future Work</strong>: I do not plan to develop this project further, but if I did I would consider the following:
<ul>
<li>Making the system more "real-time" by allowing the user to upload songs/backgrounds. The bottleneck of computation is
with the neural style computations, which can take a couple hours on a good server; however, the FFT computations for
a music file are relatively quick. </li>
<li>Interpolating frames between the keyframes generated by neural-style, to replace the current method of blending keyframes
with alpha value manipulation. Likely would use something like optical flow. </li>
<li>Exploring more sophisticated signal processing techniques so picking out treble and bass is more reliable. </li>
</ul>
</p>
</div>
</div>
</div>
<script src="readclient.js"></script>
<script src="musicvis.js"></script>
</body>
</html>