Untitled Digital Graffiti
(2023)
Concept + Installation Design
This project was done as part of a university class lectured by Prof. Tanja Diezmann.
My initial ideas when prompted on the question of future ways to interact with technology were based around how we would be using devices in the near future. Inspired by the recent developments of AR, such as the Apple Vision Pro or Meta Quest, I thought about the importance of hand control and gestures within technology. Although some usages already exist, the use of hand gestures are not very widespread throughout the world of smartphones or computers, and users always have to actually physically touch something, whether it be a screen, mouse, or controller. Knowing this, I wanted to create something more in the direction of lack of touch, and create a program or function that can be used without having to touch something or have specific equipment. My goal was for users to be able to just interact with their hands naturally in the air, and perform an action with that. Through research I found there were more advantages than comfort for the user, such as safety and hygiene - users don’t need to touch anything or risk spreading any germs in medical situations, and ease of learning - users do not need to learn how to use a specific piece of equipment or tool. I then looked up some example of gesture control in use to gain more inspiration and knowledge.
Inspired by organised graffiti walls, such as the Donaukanal in Vienna, I brainstormed possible installations combining the use of gestures with art, and eventually stumbled upon the idea of allowing anyone to "paint" on a digital graffiti wall.
I eventually stumbled upon a learning model by Google named “Mediapipe”, and it included extensive hand gesture recognition through any camera or webcam. This was the perfect adaptation of accessibility that I needed, so I chose to work with Touch Designer, a program that supported the Mediapipe plug in.
It was my first time working with the program, but I quickly learned.
My goal was to create a digital canvas in Touch Designer that would read the hand gesture controls of any person in the view of a webcam, and draw with the position of their index finger relative to the canvas. To create this, I had to do extensive research about how to use Touch Designer, as it was my first time ever using the program. My first test version included an area of a canvas only about 144 pixels wide. The program reacted to all movements of the hand and followed the index finger’s position despite where the hand was or what it was doing. Nonetheless, it functioned as a good test and gave me the understanding of how it worked. Other users also tested it, and the reception was positive as the function was super intuitive and easy to understand, people were able to understand and start drawing within seconds of even looking at the program.
Testing gave me information on what was most important to correct. - The lack of resolution, buggy hand input/tracking that was unreliable for most people, and the freedom of colour. I addressed all of these aspects and improved the program to work significantly better at a higher resolution (up to the maximum 1280x720 that Touch Designer allows). I also made it so only when the pointer finger is extended in a pointing gesture does the program draw. Finally, I made colours change proportionally to the Z axis of the hand, meaning the closer you are, the closer the hue is to 0 and the further, the closer it is to 360. This ensures that in a situation where there is a lot of space between the user and the webcam, different users’ drawings are different colours and are easily distinguishable.
The Mediapipe plugin runs very smoothly with little errors, and can detect up to 2 hands at once. The gesture recognition is also extremely effective given there is enough light on the users’ hands, which is an important limitation if the program were to exhibited in a dark environment.
The final program creates a colourful, somewhat pixelated-style look, which resembles graffiti art in a more modern form. It illustrates how the modern graffiti in the future may be digital, and how the idea of a canvas many people add to can now be in an infinite, digital space thanks to technology. The program has potential to be shown at a larger scale, and would be an effective interactive display at a large exhibition, or an area where many people may pass through, each leaving their piece of art on this digital campus. The use of hand gestures and touch-less controls also displays the effectiveness and ease of use that can be achieved, as users do not have to learn to use a new tool or equipment, and nor do they have to physically touch an object in a limited area. Instead, it shows how interaction with interfaces can be pushed to a new level in physical space around us. The project has shown me the power of gesture controls, and how this new way of interacting with interfaces is a plausible possibility in the future of technology.