Skip to content

Commit 08a987c

Browse files
author
gon
committed
README proposal 1 with images needed
1 parent e921f2b commit 08a987c

File tree

4 files changed

+42
-17
lines changed

4 files changed

+42
-17
lines changed

.bin/handlerOSC.jpeg

72.7 KB
Loading

.bin/simplified-pipeline.jpeg

124 KB
Loading

.bin/wekinator.png

99.6 KB
Loading

README.md

Lines changed: 42 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,52 @@
1-
# wekitunenator
2-
Authors: Gonzalo Nieto [@GoniGit](https://github.com/GoniGit) and Teresa Pelinski [@pelinski](https://github.com/pelinski).
3-
Wekitunenator 3000 is our final project for the Advanced Interface Design master module at Universitat Pompeu Fabra. In this project...., we built an instrument
4-
* modifies voice by appliying some FXs
5-
* it is controlled with fingers and hand positions and gestures.
6-
* FXs spaces (Fiebrink paper)
7-
* go beyond a control paradigm and use ML as an exploratory tool
1+
# Wekitunenator
2+
Authors: Gonzalo Nieto [@gonznm](https://github.com/gonznm) and Teresa Pelinski [@pelinski](https://github.com/pelinski).
83

9-
[project description + reference paper]
4+
Wekitunenator 3000 is our final project for the Advanced Interface Design course of the Sound and Music Computing Master at the Music Technology Group, in Universitat Pompeu Fabra (Barcelona). We built an interface that:
5+
* controls audio effects in real time, intended for voice.
6+
* is controlled with the user’s hand (using a camera) and a midi controller.
7+
* goes beyond a control paradigm and uses machine learning (ML) as an exploratory tool.
108

11-
(video link)
9+
Wekitunenator is inspired by [Sonami and Fiebrink’s discussion on instrument creation with ML](https://www.nime.org/proceedings/2020/nime2020_paper45.pdf). Instead of being just used to control, ML can provide access to surprise and discovery in mapping creation. Furthermore, mapping can be part of the instrument itself, not independent from it.
1210

11+
In our instrument, Wekinator maps hand positions and midi controller values to Reaper, the Digital Audio Workstation in charge of applying effects to the input signal in real time.
12+
These programs communicate through OSC messages, which are handled by a Processing sketch.
13+
14+
Therefore, there are four main parts:
15+
1. Hand tracking – made by HandPose-OSC and Processing
16+
3. OSC messages handling – done by a Processing sketch
17+
2. Mapping creation – Wekinator
18+
4. Audio and effects managing – handled by Reaper
19+
20+
<div style="text-align:center"><img src="./.bin/simplified-pipeline.jpeg"/></div>
21+
22+
(icons created by [Eucalyp](https://thenounproject.com/eucalyp/) under CCBY license)
23+
24+
Watch a short demo and explanation [here](https://youtu.be/M6bAb42s-lo).
1325

1426
## Setup
15-
[simplified pipeline]
16-
### Hand Tracking: HandPose-OSC
27+
This system has been tested in an OS X operative system.
28+
### HandPose-OSC
1729
For the hand tracking we are using [handPoseOSC](https://github.com/faaip/HandPose-OSC) by [@faiip](https://github.com/faaip/). You will need to have [nodejs](https://nodejs.org/en/) installed. First clone the repo. Then, go to the cloned repo directory `cd HandPose-Osc.git` and install the project dependencies using yarn `yarn install`. Once you've done that you can run the app with the command `yarn start`.
1830

19-
### Processing
20-
[have an executable?]
31+
### OSC handler
32+
Double click on `handlerOSC/handlerOSC.app`, which will run a java application. If HandPose-OSC is already running, you will be able to see the points of your hand –if you are showing it to your camera– in green, on top of a black background. Something like this:
33+
34+
<div style="text-align:center"><img src="./.bin/handlerOSC.jpeg"/></div>
35+
36+
Select your MIDI input device and press `UPDATE`. The FX switches are controlled by MIDI pitch indexes 0, 1, 2, 3 and 4. The REC, TRAIN and RUN buttons can be modified with MIDI pitch indexes 5, 6 and 7.
2137

2238
### Wekinator
23-
input port:8000
24-
27 inputs
39+
You can download Wekinator [here](http://www.wekinator.org/downloads/). Once downloaded, open the program and select `File>Open project...`.
40+
41+
Find the file `wekitunenator/wekProject/WekinatorProject.wekproj` and open it.
42+
43+
You will see these windows:
44+
45+
<div style="text-align:center"><img src="./.bin/wekinator.png"/></div>
46+
47+
Click on `Start listening` without changing the input port number.
48+
49+
You will be able to see that this Wekinator project receives 27 inputs –hand base points and the distances of the fingers to these points– and sends 13 outputs –five 0/1 integer values and eight floats, which control 5 effects in Reaper.
2550

26-
### Reaper configuration
27-
You can install Reaper [here](https://www.reaper.fm/). The configuration instructions can be found [here](./Reaper/README.md).
51+
### Reaper
52+
You can download Reaper [here](https://www.reaper.fm/download.php). Once it has been installed, configuration instructions can be found in this [document](./Reaper/README.md).

0 commit comments

Comments
 (0)