top | item 24799726

Pianojacq, an easy way to learn to play the piano

227 points| SanderMak | 5 years ago |jacquesmattheij.com

179 comments

order
[+] murbard2|5 years ago|reply
It's funny, I've had almost the same experience: decided to start playing again during Covid, and wanted to use the midi port to gamify things a bit. I've been using Synthesia[1] on an iPad, which also lets you take an arbitrary MIDI file and practice it.

On the off-chance that this will inspire the article's author for their development, or simply for the curious, this has been my experience with it:

The primary mode of display is "Guitar Hero" style (it's hard to describe just look at this video to get an idea[2]) but it's possible to display the sheet music. It lets you practice and grades you depending on how close you're sticking to the score. It's very nice to be able to practice one hand separately while having the app play the other hand. I do fairly well reading sheet music but the first time deciphering it can be a bit tedious, using the app speeds this up considerably for me.

Synthesia is almost great but it has a lot of small shortcomings which add up to a bad user experience, for example:

- The conversion from MIDI to a readable score is very poor compared to, say, Musescore. The sharps and flats are generally off (e.g. randomly a double flat D will appear instead of a C, even though the score has a C, Musescore interprets the midi as a C, etc).

- Not enough wiggle room when playing, e.g. if a chord is not arpeggiated in the midi file, you won't be able to move forward by arpeggiating.

- The app really depends on you following the score exactly and linearly. It should also let the score follow you. For example, if you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing.

[1] https://synthesiagame.com/

[2] https://www.youtube.com/watch?v=sleZ-hzrtRY

[+] jacquesm|5 years ago|reply
That's an interesting comment. So, the midi->score conversion is indeed one of the harder parts, I spent a lot of time on that and I'm not yet satisfied with the results. There are quite a few ways in which this can be improved further, there is a long list of 'todo's' where I basically have worked out how to make the improvement but I have not yet gotten around to actually implementing it.

Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs. You then have to do all kinds of tricks to even begin to approach what the score looked originally. Lots of enharmonically (identically sounding) combinations may have looked completely different on the original score, and when you start adding the various chord combinations it gets even more complicated. Even so, I'm strongly convinced it can be solved and we have made considerable progress on this already.

The 'wiggle room' aspect of it has been addressed to some extent. For instance, we build up a map of all the notes that are sounding at a given time, and will ignore notes that are technically 'wrong' but that are still sounding. This allows for much smoother restarts after mistakes.

Your final point is a bit tricky, but thank you for the hint, I will definitely add it to the list of things to look at to see if we can find a way to make that work.

[+] emmanueloga_|5 years ago|reply
I think what's missing on most "piano-games" is that it doesn't really mirror the way I learn a piece.

The way I learn is by bars, repeatedly re-playing bars say, 1 to 4, until I get them, them 3 to 7, until I get them, etc.. But most piano apps want you to start playing from start to end slow, then faster, etc.

So: piano apps should allow breaking a piece into bars, allow you to repeat those bars again and again until you exceed certain score. Maybe even keep track of which bars are harder to play, and repeating the difficult bars more often, etc.

[+] d883kd8|5 years ago|reply
> you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing.

I'm a piano noob but wouldn't you want to practice catching back up with the beat if you ever intend to perform?

[+] bmn__|5 years ago|reply
Notes should directly align with the keys for usability. Example: https://www.youtube.com/watch?v=6ZJTdQ-xCBE

I'm not quite happy with Synthesia. There is no visual feedback on completing a note. This has been a staple feature in Doremi Mania and Keyboard Mania for twenty years, so they have zero excuse for not implementing it.

[+] microcolonel|5 years ago|reply
Aside from the relatively difficult problem of creating scores from MIDI; when practicing piano as a beginner it'd be interesting to have help in positioning/finger selection. I wonder how hard it would be to compute fingering plans (maybe even with a wingspan parameter).
[+] bromley|5 years ago|reply
I can imagine this being really useful for improving sight-reading.

However, for me what made piano a whole lot more fun was ditching the sheet music and opting to learn about chords, all the different ways to voice them, and the various scales that fit well on top of them. I started taking lessons with a pro jazz pianist (instead of a more typical classically-focused piano teacher), I gave up on sheet music altogether and started working off lead sheets instead (just chord symbols and a melody line). I am so pleased I made that decision, it's so much more satisfying playing a tune in your own way rather than just aiming for a note-for-note reproduction of what is written on a sheet.

I still have loads more to learn, but I'm fairly pleased with where I've got to so far. I've got a few videos on youtube so you can judge for yourself if interested: https://www.youtube.com/channel/UC_NWuVoCn-Oc1KIiHEhe7Eg

However, although I think I can play reasonably well now, my sight-reading is awful, and it does hold me back because it makes it a lot harder for me to learn new techniques from sheet music (which I am actually interested in doing, as opposed to simply learning the sheet music note for note). I can do it just about, but it's painfully slow, so I am usually too lazy to bother.

Given the choice of either playing with full sheet-music or learning chords/voicings/scales and how to put them together, I'd pick the latter no question, and that is what I recommend to other adults who are learning piano, but I do think it's best to have both. So I can definitely see the value in this, even if it doesn't currently seem to teach the theory and improvisation that, in my opinion, is what really brings the joy into piano playing. I might give it a proper go myself some time soon, to see if it can help me improve my rubbish sight reading.

I can also imagine this being very useful for a beginner... Although I'm singing the praises of chord theory and improvisation I'm guessing most beginners might realistically do better starting with sheet music, for a while at least.

In summary: it might not teach everything, but it looks really useful nonetheless :)

[+] sudosteph|5 years ago|reply
I'm in a similar boat, and agree completely that it's super fun to learn this way. Coming from playing guitar, I was already used to starting from chords and listening for the rest, so it wasn't a big change. I'm a big believer in proper timing and solid rhythms being more important than notes, and learning by ear makes those aspects more intuitive, in my experience at least.

One thing I did find was the very helpful, was putting those stickers with note names on each note on the keyboard. They help me keep track of what I'm playing, and when I compose something I rely on the stickers to help me transcribe.

[+] xtiansimon|5 years ago|reply
> "I gave up on sheet music altogether and started working off lead sheets instead..."

I've not taken the plunge yet, but I was convinced this was the way to learn by Scott the Piano guy infomercial on TV. https://youtu.be/k50uk8elR0M

Some day...

[+] jacquesm|5 years ago|reply
Interesting track, there are a number of pianists (very good ones) that went that route. Keith Jarrett for instance, and Friedrich Gulda.
[+] apankrat|5 years ago|reply
Had nearly the same idea for the exact same reasons, but wanted it to work on an iPad. Safari has no WebMIDI, so I ended up using WebAudio paired with pitch detection by consensus instead. Here's the result -

https://github.com/apankrat/note-detector

It's not without issues still, but tends to do a reasonably good job at what it aims to do.

[+] jacquesm|5 years ago|reply
Oh, neat! That's very nice, we were actually discussing doing this the other day using FFTs to make it work with real pianos that do not have a silent option installed.
[+] rednum|5 years ago|reply
This seems a bit similar to Melodics[1], except open source and it actually uses the standard notation.

I know how to read music and started taking classical lessons as an adult. Some time last year a friend recommended me to check out Melodics. It has 3 instruments: keys, pads (finger drumming), drums. I've played pads for few mont;hs, that was good, I've felt it improved my timing and rhythm skills quite a bit. Then I switched to keys. I didn't really like that; it felt like I'm mindlessly repeating whatever is on the screen instead of internalising music. OTOH, when playing pads it felt that something stays in my head after playing. So I guess this may be useful to some people.

For me it'd be more interesting to have some sort of memory trainer - I play something on screen few times, then I replay from memory and app shows me a diff maybe? Or when I'm learning a piece, it would give me a challenge "play bars 21-25 both hands" or "play bars 19-23 left hand only".

[1] https://melodics.com/, I'm not affiliated in any way

[+] DePro99|5 years ago|reply
This is just what I needed. I just bought an electric keyboard last week and have practiced with Melodics every day since (the free version, so 5 minutes every day :P), but I have some issues with things like how it just keeps going if you mess up or are too slow and the fact that it doesn't use sheet music or let you use your own music (granted, I haven't looked into the subscription version's features yet).

I just opened this and spent 20 minutes trying to play 4 bars I wrote a year ago in MuseScore. I am definitely a noob at playing, hahaha. I always figured being able to play in real-time isn't of much value, but I'd like to be able to improvise and use a piano to enter notes when composing at least, and then I saw some of Lionel Yu's videos and bought a keyboard... (https://www.youtube.com/c/LionelYuPiano/videos)

I definitely appreciate the auto-replay, simultaneous note count limit, waiting until you hit a key (both at the start and mid-song if you're slow), and scoring.

[+] jacquesm|5 years ago|reply
> Or when I'm learning a piece, it would give me a challenge "play bars 21-25 both hands" or "play bars 19-23 left hand only".

It does exactly that, pick 'auto' mode.

[+] LegitShady|5 years ago|reply
from the last paragraph of the above link

>The software is not ‘open source’, we have not yet decided on how we plan to go forward with this project in the longer term, but the code is on GitLab and if you want to see what makes it tick or suggest improvements then of course you are more than welcome to do so.

[+] vram22|5 years ago|reply
Ha ha, good one, Jacques.

Another piano app, by me, this is as rudimentary as one can get, in Python. It turns your PC keyboard into a "piano":

Play the piano on your computer with Python:

https://jugad2.blogspot.com/2013/04/play-piano-on-your-compu...

The comments by people wih more music knowledge than me (zero) were interesting and informative.

[+] TedDoesntTalk|5 years ago|reply
> Unfortunately, because Mozilla is too busy improving the world instead of their browser and WebMIDI is not supported on Firefox you have to use Chrome

Attention Mozilla

[+] jl2718|5 years ago|reply
I did a lot of tab in my youth, but got nowhere with it; I think because I didn’t understand theory like chords, scales etc, and when exposed, I never recognized them. Ear players seem to be able to chord along to just about anything and then add detail until it resembles the original. There must be some way to help the learner recognize the theory while teaching rote.
[+] musicale|5 years ago|reply
> Note that it would have been trivial for us to force our users into a relationship with us by having to make an account and we made a very conscious choice not to do this.

Bravo! Well done and well said - I really wish more projects and systems were developed with this mindset, viewing users as people to be empowered rather than as resources to be captured and exploited.

[+] wazoox|5 years ago|reply
That's an interesting tool. As a former professional pianist, I'd repeat what I feel like the limitation of such a program: it does teach you note reading, which is nice, but really basic piano technique is a necessary preamble. You mentioned that your boy spent a whole day on this; with a bit of guidance, a full beginner after a few hours should be able to play this very simple melody with one hand, and to add a couple of bass notes with the other, for a much more satisfying result (for the player and for the audience). I know it because I've done it with many children in the past.

So you should probably add something like some introduction material / tutorial... maybe a couple of videos.

[+] jacquesm|5 years ago|reply
Yes, this is an urgent need and recognized. We are in the process of hiring a piano teacher to help with this as well as with making the program more absolute beginner friendly.

This project is a lot of work as it is today, it was made mostly to satisfy a need that I felt myself and my colleagues picked it up and ran with it so now it has some proper power behind it. We fully intend to 'stay the course' and make this a best-in-class offer but there is a long way to go before we can claim that we have achieved that.

As for the satisfaction, I think you may be off the mark there, I've had enough piano education to know what that was like and Luca is having a lot more fun with this program than I ever did being taught the piano by teachers at roughly the same age.

[+] kej|5 years ago|reply
I understand the issue with Firefox not supporting WebMIDI, but it would be nice if instead of redirecting to a separate page you could show the same message as a warning on the main page. Let me see the app and see that it's worth coming back in Chrome.
[+] jacquesm|5 years ago|reply
Good point, thank you, I will add this to the todo list.
[+] rzodkiew|5 years ago|reply
Regrettably it's not working on Firefox[1], but hey at least I can report wrongthink YouTube videos.

[1] https://pianojacq.com/firefox.html

[+] jacquesm|5 years ago|reply
Yes, very much regrettably, I now find myself using Chrome when I really would like to avoid that. But unfortunately there is no way around this right now that I've found to be reliable enough and easy enough to the point that I would advise users to go that route. Sorry.

Note this 8(!) year old bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=836897

It's embarrassing.

[+] drchopchop|5 years ago|reply
This may teach you to "play" piano, but the real question is whether this sort of rote memorization will really teach you how music fundamentally works. Seems a bit like re-typing Javascript programs in order to "learn" programming?

Will you be able to improvise after enough of this? Play songs by ear?

There are a few fundamental musical concepts that I'm not sure you easily pick up by rote memorization:

* Most music is based around "chords", which have non-obvious mappings to physical keys. (They're linear intervals in frequency space, but that doesn't make it easy to realize that D/F#/A and Ab/C/Eb are two representations of essentially the same thing)

* Likewise, songs can be transposed up or down without affecting their fundamental attributes. (They just sound "higher" or "lower"). However, to do that, you need to be able to shift +/- N keys which is tricky given the difficult layout of a piano. To really do this well, you need to understand the concept of scales/modes.

[+] save_ferris|5 years ago|reply
> Will you be able to improvise after enough of this? Play songs by ear?

How many Herbie Hancock's have you come across after finishing a grade 1 piano book? Of course a single exercise app or book isn't going to create the perfectly rounded musician. For a child or beginner player, this looks like a great tool.

Advanced musical skill is entirely composable. Having solid improvisational skill or technical ability largely depends on how much you practice those various facets of your instrument. The reality is that probably 95%+ of people who pick up an instrument in their lives never make it to that point anyway.

You're trying to critique this tool with the expectation that a single learning resource should address all aspects of playing an instrument, when that's never been true of any musical learning resource ever.

[+] jacquesm|5 years ago|reply
No, it teaches you how to read the scores pretty good actually. The 'rote memorization' can be pushed as well but that's not the goal here, the goal is to learn to read notes fast enough that you can actually play the music. (See attached video).

As for improvisation and playing songs by ear: improv requires some knowledge about how music is structured, I have some ideas on showing what the underlying structure is but these have not yet been implemented, we are for now concentrating on accuracy of the rendering of the score (which is still plenty of work) and fingering.

Playing by ear is mostly to have a good association between the keys and the sounds they make, this gets harder with chords. As with everything, practice makes perfect and doing this frequently seems to help me quite a bit by picking out songs by ear. So even if it isn't a direct goal of any of the modes of operation of the software it does seem to be a side effect. We will add an 'ear training' module at some point.

[+] tekkk|5 years ago|reply
I'd wager it can't teach you abstract concepts but since abstract concepts are much easier to learn with the practical knowledge of using them, it would definitely be helpful. Also honing motoric skills is a key part in doing many things - arts, sports - so getting accustomed to that is a big plus too. The downside of doing only that is yes, you will be quite lost with the abstract and also reinforce some bad habits.

So no cynical HN commentary from me! If it works, it is definitely a big help for learning to play piano.

[+] onion2k|5 years ago|reply
Not everyone wants to become a composer. A lot of people just want to be able to sit down at a piano and play music they enjoy. Learning by rote works very well for that.
[+] turtlebits|5 years ago|reply
This is great! My son, who normally loses interest in long songs that are a bit harder than his level, got through The well tempered clavier (one of the sample songs).

The timing/spacing of the notes is a bit off though, I wish it scrolled according to beats instead of speeding up and down depending on what type of note was being played. I understand it has to stop and wait for the note to be played, but some of the whole notes threw me off.

[+] prionassembly|5 years ago|reply
My dad is a music tutor, piano and guitar.

He does two different "tracks" according to what the student wants: (1) learn currently popular songs or (2) learn didactic music in the classical tradition (e.g. Czerny). In either track he's really teaching music theory: most popular music will be I-IV-V; jazzy music with turnarounds do ii-V-I tricks, etc. He doesn't teach an instrument, he teaches music.

[+] jacquesm|5 years ago|reply
Very nice! Is your dad available for a consultation?

We would very much like to teach things properly but for that we need more understanding and someone who has been teaching music would be a huge asset to us.

If you could please ask that would be awesome.

[+] person_of_color|5 years ago|reply
There’s a cottage industry of piano learning apps, and an equivalent amount of people getting frustrated at the lack of progress and quitting
[+] jacquesm|5 years ago|reply
That's why we spent quite a bit of time on making interesting metrics that show your progress over time, even if it is small per increment the overall effect is substantial.
[+] drran|5 years ago|reply
I had good experience with Android app made by soft²cat, but it long gone. Unlike other guitar-hero like apps, it waits for the note to be pressed, so I cannot miss it, and the app discards the note when I press the note earlier. It allows me to play melodies at my own tempo: slower or faster than the original, which is much easier for me.
[+] khazhoux|5 years ago|reply
The sadness of my piano life is not my fingers but my dumb brain. No matter how intensely I practice and memorize a piece, if I step away from it for a week I'll forget 30% of it.

Curiously, my audio memory is very strong. I can hear music in my head clearly and in detail, even songs and pieces I haven't heard in years. This does not translate to "finger-memory".

[+] jacquesm|5 years ago|reply
I'm going to guess here and say that you are practicing a very limited repertoire, rather than to learn to play the instrument, in which case you are memorizing movements instead of music.

Is that correct?

[+] quacked|5 years ago|reply
When you practice and learn a piece, how exactly to you practice and memorize it?
[+] weinzierl|5 years ago|reply
I had the opportunity to beta test it and while it still has rough edges my daughter and me had a lot of fun with it. Everyone learns differently but in my experience direct feedback is often such a boost in learning speed and efficiency that I wish there were more applications like this. Also the attitude towards user privacy is highly appreciated.
[+] pianoboard|5 years ago|reply
On this note,seeing as cheap qwerty keyboards easily come in $5 and good in $20, is there a "piano" keyboard that only has keys laid out in piano fashion, rest left to drivers a small software that plays appropriate tones ....

I'd prefer something small- only two octaves and a plus/minus button to shift octaves.

[+] jacquesm|5 years ago|reply
I will look at using the regular PC keyboard as a stand-in midi keyboard of ~ 2 octaves. This is not going to give a great experience (especially because many keyboards have limited key-rollover) but it should give a less-hardware-required onboarding experience, which is always a good thing.
[+] mattstudio|5 years ago|reply
You may want to check out https://synthesiagame.com/ which is a commercial product that does the same thing. You don’t have to use the guitar hero style of notes, you can turn on proper sheet music.
[+] dangerboysteve|5 years ago|reply
This is good timing. I was thinking about taking up the piano for fun. My plan is to rent a keyboard for a month (Roland FP10/FP30) and going the YouTube route to see if I liked it.