ARE YOU FINDING GAMES TO HARD?
DO YOU STRUGGLE WITH ALL THOSE EXTRA KEYS?
DO YOU WANT TO FEEL SPIRITUALLY CONNECTED TO YOUR SCREEN?
Well do we have the device for you with crazy controllers.
Crazy controllers presents two unique ways of controlling the game.
Inspired by the Nintendo Power Glove we created a gauntlet that is able to recognise different gestures based on the position of your hand using an artificial neural network. Allowing you to control your computer and games in fun, unique ways.
We used potentiometer analog sliders and an IMU to read the current state of the hand position, the IMU providing the detail on the orientation of the hand using its accelerometer and using the analogue sliders connected to strings in antagonistic pairs to find out the extension of each finger. These sensor values are then sent along the serial monitor and parsed by a python script running on the computer which uses them to predict the current gesture based on a large csv file contianing sensor values and their corresponding gestures.
The glove uses an artificial neural network built in tensorflow to classify between the different gestures based on the sensor values, you can also add your own gestures and assign them to computer inputs directly though the python gui.
Complementing the gloves, a more portable companion was developed. We went on a little sidequest to also build this cute cube controller using an ESP32 microcontroller, and connected it to a generic joystick breakout module and a 1.8" 160x128 TFT Display. The joystick is controllable in both X and Y axes and has a switch you can click, similar to thumbsticks you'd find on modern controllers.
Getting this to work was a bit tricky since ESP32s don't do USB HID natively (the protocol for controller input), so we experimented with serial feeding via IBus to send the joystick data over serial and used vJoy and vJoySerialFeeder to turn it into a virtual joystick controller that a laptop recognises.
Calibrating the joysticks was super annoying, they never centre exactly and constantly drift (it drifts just by looking at it T-T), so we mapped readings to between 1000 and 2000 (~1500 centre, with a deadzone). The screen displays uptime and a cool live joystick visualiser (perfect for crying at the sight of stick drift 😭).
For an individual demo of the joystick, see this video where I tested it out on my game for Juice (Hack Club) :).
