Issuu

AirKey

Interview "Teaching music is my passion, and I want to give my students the best possible education. But it's disheartening to see them struggle due to the lack of resources."

#Hands Gesture recognition #Arduino

" I love music, especially playing the piano, but it's so hard to pursue my passion here. I wish there was a way for me to learn and practice without the financial burden. " Davide. A music major graduate Demographics: Age: 22, Male, College student.

Emily. A music teacher

Individual project Hardware Design/UX Design

Pain Points

Demographics: Age 32, Female, Lives in a suburban area, Limited school budget School music teacher.

for music instruments Difficulty in engaging Goals: To make music education more accessible and students with varying engaging for her students, especially those who might find interest levels in music. traditional instruments intimidating or uninteresting.

YEAR：2022.12 - 2023.3

Pain Points

ackgrounds＆Research

ack of Access: Limited access to musical instruments due to financial constraints and remote location Inadequate Music Programs: School's music program is underfunded, with few opportunities for hands-on learning. L

The gift of music belongs to everyone, yet many lack access to instruments that make learning possible. Between tight budgets and remote locations, children from poverty-striken communities in Asia are often barred from experiencing music's brilliance.

Expenditure

2000

79.5%

Hand Detection and Finger Counting Using OpenCV-Python

Ultrasonic Air Piano Hand Detection by 3D hand gesture recognition model

This tutorial extends the article on "Ultrasonic Parking Sensors" and utilizes the ultrasonic sensor of the micro:bit self-driving car (MoonCar Lunar Rover) to detect distances. These distances are then converted into piano scales, allowing for the playing of piano tunes through ultrasonic waves, essentially enabling the performance of piano melodies in the air.

TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing

Convexity Convex Hull

Defect

500

Contour

Workshop

Communication and Coordination

udget vs Expenditure

ehman, P. R. (2020). A Look Ahead: Music Education from 2020 to 2050. Contributions to Music Education, 45, 67-80. Retrieved from ERIC - EJ1256014.

Inspiarions

1500

Resource Development

Disbursement of Funds

Pain Points

https://steam.oxxostudio.tw/category/microbit/mooncar/ultrasonic-piano.html

1000

The disbursement chart illustrates a prudent allocation of resources, with a significant portion of funds effectively utilized, ensuring the project's sustainability and impact.

oals: Look forward to the possibility of learning, creating, and sharing music without the need for expensive instruments.

oals: help students to overcome barriers and engage in music.

By importing Libraries- cv2 and numpy for handling arrays as well as for math, we can implement hand detection.

2500

Amount in USD

20.5%

Demographics: Age: 19, Female, graduate student.

9 M

Design Thoughts Budget

3000

Jia. A music major graduate

Ideation

Airkey is our vision to remove barriers to music education. Using hand gesture recognition, Air Piano gives users the power to produce beautiful piano melodies using just their hands and the air. Without expensive equipment required, Air Piano allows the joys of musical creation to exist anywhere.

The bubble chart reveals efficient budget management, with expenditures closely aligning with the planned budget, emphasizing our project's cost-effectiveness in delivering educational value.

piano tutor.

Difficulty in finding a clear path within the diverse music world Fear of complex music concepts and traditional instruments Limited access to musical resources, impacting exploration and growth.

"I've always dreamed of playing the piano."

I've always loved music, especially playing the piano, but it's so hard to pursue my passion here. I wish there was a way for me to learn and practice without the financial burden. Shen. A piano tutor Demographics Age 3 , ale, :

oals: Davide wants to enjoy his journey into music. He wants music to be a source of fun, creativity, and self-expression.

Pain Points

TapID, a wrist-based inertial sensing system that complements headset-tracked hand poses to trigger input in VR. TapID embeds a pair of inertial sensors in a flexible strap, one at either side of the wrist; from the combination of registered signals, TapID reliably detects surface touch events and, more importantly, identifies the finger used for touch. . eier, P. Streli, A. Fender and C. Holz, "TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing," 2021 IEEE Virtual Reality and 3D User Inter faces (VR), Lisboa, Portugal, 2021, pp. 519-528, doi: 10.1109/VR50410.2021.00076. M M

iterature Review

The current music education system is not widespread or inclusive enough, leading to a loss of creative expression and cultural diversity opportunities.

igh costs and lack of resources limit access to music education for many, particularly in underfunded schools and low-income families.

Socio-economic Barriers

Systemic Inequality in Music Education

daptation to Student Needs

The financial aspect is one of the biggest challenges in music education. Many students and schools struggle with the high costs associated with learning and teaching music, including the expenses for instruments, sheet music, and quality teaching. This financial barrier often prevents students from accessing quality music education

emote areas face geographical isolation, and there is a scarcity of trained instructors, hindering the Geographical and opportunity for musical talent Instructional Challenges development. R

Systemic Problems Curriculum and Pedagogy Weaknesses There's a general consensus that music education often suffers from weaknesses in curriculum and pedagogy, which may be due to inadequate training, lack of support, and insufficient access to innovative ideas. This issue is exacerbated by a lack of creative and interactive group work in music education, leading to a decline in creative expression and practical skills among students.

inancial Challenges

s r e i r r a B c i m o on

ack of Motivation and Commitment

Many students face challenges in commitment and motivation. Students often fail to understand the depth of commitment required to master musical skills, and motivation can be difficult to sustain, especially when the student's interest in music is not self-driven.

Problem Statement Problem Solution Strategy

ccessibility to music education is limited by socio-economic barriers, with traditional learning requiring instruments that are often too costly A

The high cost of musical instruments, lack of trained instructors, and geographical isolation hinder the nurturing of musical talent Inequality in music education leads to a loss of opportunities for creative expression and cultural representation, exacerbating educational and cultural disparities. Technological Disruption Approach: Develop and implement disruptive tech solutions (like virtual instruments) that redefine the means of learning music Outcome: Make music education more affordable and accessible.

Community-Centric Models Strategy: Tailor music education initiatives to fit the unique cultural and social fabric of each community Impact: Foster local engagement and support, making music education a communal priority.

Empowerment through Creativity

Cultural and Educational Metrics

Aspiration: Empower individuals to explore and express their cultural identities through music Long-Term Goal: Cultivate a global community where creative expression through music is a universal right.

Evaluation: Use both qualitative and quantitative metrics to assess the impact on cultural representation and educational access Adjustment: Continuously refine strategies based on feedback and outcomes.

Design flo

Wiring Diagram

P H d

54mm

Start

Gesture Recognition

Output result

Sensors Setup

ers

hand gesture data 3D Gesture Recognition Sensor x1

7 Du

2mm

ont

lines

O x1

arious

Function Description of gesture sensor

appropriate musical Analyze Gesture

Output result

signal from the processed

Data in Python

sol

gestures.

ering iron x1

electric sol

USB Cable x1

ering pen

GND

Negative electrode

Working voltage: 3.3-5

VCC

Positive electrode (5V)

Interface type: I2

MCLR

Sensor reset (low level valid)

I2C address: 0x4

SCLI2C Clock signal

Gesture sensing: 5c

SDAI2C Data signal

Distance sensing: 10c

Operating temperature

Digital IO port

range: -20°C ~ +85° Spatial resolution: 150 dpi

Sensor address: 0x42

Data Transmission

Output result Outputting

from Arduino to audio output device

Music

Ar Git

uino co

ub: natalilfis

d - y e p

Gesture note correspon

h d

pie23/pro ect2_airkey_

gesture_recognition

( h . git

ub com

)

For piano music creation

ence

we can map different gestures to specific piano notes or scales.

#Sensor Initialization myGesture = Over

DFRobot_MGC3130(d_pin=20 mclr_pin=21

Right

us=1)

Selects a suc

#Setup Function

Components Assemble

h h 4 ig

as G

er single note

def setup():

ile(myGesture.begin() != True):

print("Begin err!!! Please c

eck w

Clockwise Maps to an ascending

h h et

er t

melody or scale

connection is correct")

specifically

e C ma or scale.

print("begin success")

ile(myGesture.enable_gestures() != 0):

print("enable gestures err")

print("enable gestures success")

#Loop Function def loop():

Down

ascending scale on t

myGesture.sensor_data_recv()

( 4) C

if(info == myGesture.FILCK_R):

print("Flick Left to Rig

, h dd h h ( 5).

e piano

as from mi

info = myGesture.get_gesture_info()

Corresponds to a

suc

descending scale

le C

on t

4 6

t to Left")

print("Flick Down to Up")

Left Selects a lower

Anticlockwise

elif(info == myGesture.FILCK_U):

single note

suc

as G3.

Maps to a descending

elif(info == myGesture.FILCK_D):

melody or scale

scale

elif(info == myGesture.CIRCLE_CLOCKWISE):

print("Circle clockwise")

d d ite

S Co

d 2020 e

roug

h h t

is met

scales by gestures

h A

specifically t

print("Flick Up to Down")

print("Circle counterclockwise")

( 4) C

elif(info == myGesture.CIRCLE_COUNTERCLOCKWISE):

, dd w (

e piano

as from mi

h ")

elif(info == myGesture.FILCK_L):

print("Flick Rig

Corresponds to an

minor

users can control and play different piano notes or

creating a variety of musical effects.

to lo

suc

le C

Insights

Future Scenarios

Initial Learning Curve

Some users' initial struggles with precise gesture control underscored the need for a more guided learning approach at the beginning stages. Adapting to User Needs

Music Therapy for the Elderly: Elderly individuals in a nursing home enjoying music therapy with AirKey. Public Installations: An interactive AirKey installation in a public park.

Usability Test Student, 22

Collaborative Music Projects in Schools: Students collaboratively creating music in a school setting.

The suggestion for a progressive learning system highlighted the importance of adapting the product to accommodate various skill levels.

Feedback Mechanisms

Special Education and Therapy: Students with disabilities using AirKey in a classroom.

Experienced piano player

Music teacher

"The gesture recognition is good, but at times it doesn’t capture the nuances of a real piano."

"It’s a fantastic tool for introduction to music. However, I would love more structured lessons integrated into the system."

"It makes me feel like a real musician!"

The test emphasized the necessity of incorporating simple, optional feedback mechanisms to aid the learning process. Progressive Learning Features

There's a clear need for a structured, progressive learning program that starts with basic notes and melodies.

Student,22 "There were moments when it didn’t quite catch my hand movements, especially when I moved fast."

"More guidance or interactive tutorials for beginners would be beneficial."

Parents

Future Improvements for Airke

Expanding Musical Range: Future iterations could include a wider range of instruments and sounds, enabling users to explore different genres and styles of music Interactive Learning Modules: Incorporating educational tools and interactive learning modules within the Airkey system could enhance the learning experience, offering guided lessons and progress tracking Sustainability and Affordability: Ensuring the technology remains affordable and sustainable, particularly in terms of energy consumption and device longevity, will be crucial for widespread adoption.