|
1 |
| -# wekitunenator |
2 |
| -Authors: Gonzalo Nieto [@GoniGit](https://github.com/GoniGit) and Teresa Pelinski [@pelinski](https://github.com/pelinski). |
3 |
| -Wekitunenator 3000 is our final project for the Advanced Interface Design master module at Universitat Pompeu Fabra. In this project...., we built an instrument |
4 |
| -* modifies voice by appliying some FXs |
5 |
| -* it is controlled with fingers and hand positions and gestures. |
6 |
| -* FXs spaces (Fiebrink paper) |
7 |
| -* go beyond a control paradigm and use ML as an exploratory tool |
| 1 | +# Wekitunenator |
| 2 | +Authors: Gonzalo Nieto [@gonznm](https://github.com/gonznm) and Teresa Pelinski [@pelinski](https://github.com/pelinski). |
8 | 3 |
|
9 |
| -[project description + reference paper] |
| 4 | +Wekitunenator 3000 is our final project for the Advanced Interface Design course of the Sound and Music Computing Master at the Music Technology Group, in Universitat Pompeu Fabra (Barcelona). We built an interface that: |
| 5 | +* controls audio effects in real time, intended for voice. |
| 6 | +* is controlled with the user’s hand (using a camera) and a midi controller. |
| 7 | +* goes beyond a control paradigm and uses machine learning (ML) as an exploratory tool. |
10 | 8 |
|
11 |
| -(video link) |
| 9 | +Wekitunenator is inspired by [Sonami and Fiebrink’s discussion on instrument creation with ML](https://www.nime.org/proceedings/2020/nime2020_paper45.pdf). Instead of being just used to control, ML can provide access to surprise and discovery in mapping creation. Furthermore, mapping can be part of the instrument itself, not independent from it. |
12 | 10 |
|
| 11 | +In our instrument, Wekinator maps hand positions and midi controller values to Reaper, the Digital Audio Workstation in charge of applying effects to the input signal in real time. |
| 12 | +These programs communicate through OSC messages, which are handled by a Processing sketch. |
| 13 | + |
| 14 | +Therefore, there are four main parts: |
| 15 | +1. Hand tracking – made by HandPose-OSC and Processing |
| 16 | +3. OSC messages handling – done by a Processing sketch |
| 17 | +2. Mapping creation – Wekinator |
| 18 | +4. Audio and effects managing – handled by Reaper |
| 19 | + |
| 20 | +<div style="text-align:center"><img src="./.bin/simplified-pipeline.jpeg"/></div> |
| 21 | + |
| 22 | +(icons created by [Eucalyp](https://thenounproject.com/eucalyp/) under CCBY license) |
| 23 | + |
| 24 | +Watch a short demo and explanation [here](https://youtu.be/M6bAb42s-lo). |
13 | 25 |
|
14 | 26 | ## Setup
|
15 |
| -[simplified pipeline] |
16 |
| -### Hand Tracking: HandPose-OSC |
| 27 | +This system has been tested in an OS X operative system. |
| 28 | +### HandPose-OSC |
17 | 29 | For the hand tracking we are using [handPoseOSC](https://github.com/faaip/HandPose-OSC) by [@faiip](https://github.com/faaip/). You will need to have [nodejs](https://nodejs.org/en/) installed. First clone the repo. Then, go to the cloned repo directory `cd HandPose-Osc.git` and install the project dependencies using yarn `yarn install`. Once you've done that you can run the app with the command `yarn start`.
|
18 | 30 |
|
19 |
| -### Processing |
20 |
| -[have an executable?] |
| 31 | +### OSC handler |
| 32 | +Double click on `handlerOSC/handlerOSC.app`, which will run a java application. If HandPose-OSC is already running, you will be able to see the points of your hand –if you are showing it to your camera– in green, on top of a black background. Something like this: |
| 33 | + |
| 34 | +<div style="text-align:center"><img src="./.bin/handlerOSC.jpeg"/></div> |
| 35 | + |
| 36 | +Select your MIDI input device and press `UPDATE`. The FX switches are controlled by MIDI pitch indexes 0, 1, 2, 3 and 4. The REC, TRAIN and RUN buttons can be modified with MIDI pitch indexes 5, 6 and 7. |
21 | 37 |
|
22 | 38 | ### Wekinator
|
23 |
| -input port:8000 |
24 |
| -27 inputs |
| 39 | +You can download Wekinator [here](http://www.wekinator.org/downloads/). Once downloaded, open the program and select `File>Open project...`. |
| 40 | + |
| 41 | +Find the file `wekitunenator/wekProject/WekinatorProject.wekproj` and open it. |
| 42 | + |
| 43 | +You will see these windows: |
| 44 | + |
| 45 | +<div style="text-align:center"><img src="./.bin/wekinator.png"/></div> |
| 46 | + |
| 47 | +Click on `Start listening` without changing the input port number. |
| 48 | + |
| 49 | +You will be able to see that this Wekinator project receives 27 inputs –hand base points and the distances of the fingers to these points– and sends 13 outputs –five 0/1 integer values and eight floats, which control 5 effects in Reaper. |
25 | 50 |
|
26 |
| -### Reaper configuration |
27 |
| -You can install Reaper [here](https://www.reaper.fm/). The configuration instructions can be found [here](./Reaper/README.md). |
| 51 | +### Reaper |
| 52 | +You can download Reaper [here](https://www.reaper.fm/download.php). Once it has been installed, configuration instructions can be found in this [document](./Reaper/README.md). |
0 commit comments