JohnVinyard.github.io/rnn-instrument.html at master · JohnVinyard/JohnVinyard.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="preconnect" href="https://fonts.googleapis.com" />
    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
    <link
      href="https://fonts.googleapis.com/css2?family=Roboto:ital,wght@0,100..900;1,100..900&display=swap"
      rel="stylesheet"
    />

    <title>RNN Musical Instrument</title>
    <script src="build/components/bundle.js"></script>

    <style>
      body {
        font-family: "Roboto", sans-serif;
        background-color: #222;
        color: #eee;
        line-height: 1.5em;
        font-size: 1.2em;
      }

      h1 {
        font-size: 2em;
      }

      hr {
        display: block;
      }

      .container {
        display: flex;
        flex-direction: column;
        align-items: center;
        align-content: center;
        margin-right: 10vw;
        margin-left: 10vw;
      }RNN Musical InstrumentRNN Musical Instrument

      video {
        width: 90vw;
      }
    </style>
    <template id="demo-template">
      <div>
        <instrument-element
          url="https://state-space-model-demo-3.s3.amazonaws.com/rnnweights1_5362df743be8105f6de3e0f664c899fca81c5f82"
        >
        </instrument-element>
      </div>
    </template>

    <script>
      document.addEventListener("DOMContentLoaded", () => {
        const tryItLink = document.getElementById("try-it-link");

        tryItLink.addEventListener("click", () => {
          const tmp = document.getElementById("demo-template");
          const container = document.getElementById("demo-container");
          const clone = document.importNode(tmp.content, true);
          container.appendChild(clone);

          const video = document.querySelector("video");
          video.remove();

          const tryItLink = document.getElementById("try-it-link");
          tryItLink.style.display = "none";

          container.style.height = "600px";
        });
      });
    </script>
  </head>

  <body>
    <section class="container">
      <section class="item">
        <h1>Recurrent Neural Network Instrument</h1>
      </section>
      <section class="item" id="demo-placeholder">
        <video
          preload="metadata"
          src="https://state-space-model-demo-3.s3.us-east-1.amazonaws.com/rnn-instr-demo.mp4#t=14.5"
          height="500"
          width="500"
          controls
        ></video>
      </section>
      <section class="item">
        <a id="try-it-link" href="#demo-anchor"
          ><h3>Try It (requires webcam permissions)</h3></a
        >
      </section>
      <section class="item" id="demo-container">
        <a id="demo-anchor"></a>
      </section>
      <section class="item">
        <h2>About</h2>
        <p>
          First, we overfit a single-layer RNN, factoring a short segment of
          classical music into two components:
        </p>
        <ul>
          <li>a sparse "control-signal"</li>
          <li>
            a recurrent network that models the resonances of the instrument(s)
            being played.
          </li>
        </ul>
        <p>
          Then we map MediaPipe
          <a
            href="https://mediapipe.readthedocs.io/en/latest/solutions/hands.html#hand-landmark-model"
            >hand-tracking landmarks</a
          >
          into the RNN's input space so that the learned network can be "played"
          via hand-movement.
        </p>
        <p>
          Training details can be explored further
          <a href="https://blog.cochlea.xyz/rnn.html">here</a>
        </p>
      </section>
    </section>
  </body>
</html>