This is a talking piano and it's so cool and his name is chopsticks. So in addition to talking, he could play simple songs like that, but also like really complicated songs. So today, not only will we talk about the science behind making a piano speak human, but later he's gon na play the drums for us and sing for us not yet chopsticks. Remember for my videos that song usually comes a little bit later.

Anyways then, he's gon na attempt to play the world's most complicated digital piano song from sheet music boss rush e there's a reason. This song has never been played with an actual real-life piano, so i've legitimately got a couple fire extinguishers on hand in case the whole thing goes up in flames, but before we get to any of that, we need to first understand the really cool engineering behind how Pianos even work, because if you look at the guts of this modified see-through piano for something that was invented by an italian dude 300 years ago, you can see it's pretty complicated. In fact, if you zoom in and just isolate on a single key, this is what's happening every time you push down on the white part. Now, if you're thinking like an engineer, you know the best design is always the one that gets the job done in the most simple way possible, and when i first saw this no offense to bartolome christofori, the inventor of the piano, it just seemed overly complicated.

I mean the first piano i ever played was pretty dang simple like. Why can't you just have a class one lever where, when i push down here, there's a full crimp in the middle and it hits the string here well, it turns out there's two problems with that right off the bat. The first is that the hammer remains in contact with the string when you hold the key down. That means the string isn't free to wiggle and vibrate, which is how the sound is created like with this guitar.

It sounds great until you touch the strings and now because it can't vibrate, you don't hear anything. In fact, you can see that's exactly the problem with this toy piano. The second problem is, this is an upright piano, so the strings must be vertical to fit, but that's an easy enough fix if we shift them here and add an extra hinge point. However, you'll notice we're still damping the string and, as you can see, the piano should continue making sound, even if i'm holding the key down.

So how do we do that? Well, bartolomeo got real, clever and added a little extra hardware here. So now, when you press the key this jack flings the hammer forward, so it hits the string, but then the jack gets its foot caught which moves it out of the way. So the hammer can move back and not dampen the string fun fact. Most people think pianos are considered string instruments, but this hammer-like action is why they're actually classified as percussion okay.

So now, if i press the key down, it will continue to make noise but wait. If i let my finger go, the sound should stop. So something is damping that string. But if you look in our current build, you can see if i let the key go, there's nothing to stop the string from just continuing to vibrate and make sound to solve this bartolomeo added an extra pivot point and some more hardware.

So now you see this damper rests on the string and keeps it quiet until you press the key down, then the hammer strikes, then it's free to vibrate and create noise all the way up to the moment the key is released. And finally, if i want to play note really quickly and repeatedly like this, i need the hammer to be close to the string and not way back here like this. So by adding this backstop, it keeps the hammer close after each hit in case. I need to quickly hit it again, so there you have it apologies to my fellow engineer, bartolomeo.

It turns out it's only as complicated as it needs to be, and no more, it's also sort of simplistically beautiful that all that functionality is from only one input. Every time you press a single key now, chopsticks here is a special case, not only because he's see-through, but because he can play himself, and it is true pianos that play themselves have been around for over 100 years, but there's an important difference here. An old player piano works by scrolling through a paper sheet like this and when it encounters a hole it passes, air through which plays the key. So all the keys are played at full force and the timing isn't very precise, whereas chopsticks here is a modern version that uses a solenoid to actuate the key we place the solenoids right here so when the rod extends it's no different than someone actually pressing.

The key you can see them all lined up along here, but with solenoids. Not only do you get your timing down to fractions of a second, but you have 127 different levels of increasing force to press a piano key, which means you can make perfect recreations of a human player. So here's my friend andrew from the youtube channel sheet music boss playing a simple c shanty and then recording it as a midi and then chopsticks uses that midi to play it back perfectly. Only since he's not limited to a mere 10 human fingers, the songs can get more complicated, but he can do even better than that, because those 88 precision solenoids allow him to make crude reproductions of entire bands, see if he can pick out the singer's actual voice.

In this classical piece of music history, and so now that we know at least partially what he's capable of before we attempt rushi, let's quickly discuss how we actually make chopsticks speak and sing chopsticks, that's better, and to do that. I'm gon na pass it over to my buddy grant from the youtube channel three blue one brown for a simple explanation. The signals from speech can be visualized with a waveform which you might think of as telling you the air pressure next to a microphone as a function of time. If you zoom in on a little window of it, you might notice.

It looks like a rhythmic repeating pattern. As it happens, one of the most delightful facts from math tells us that for pretty much any signal, you can express it as a sum of pure sine waves which, in the context of sounds, correspond to pure pitches, higher frequency sine waves, giving you higher pitches. If you want to perfectly reconstruct the original signal, this often requires adding a very large number of pure sine waves, each with a different height. Now a favorite tool in signal processing is something called a fourier transform, which you can think of as telling you the heights of each one of these sine waves as a function of the frequency you're dealing with so for a piano project.

If you take this fourier transform and you just consider the peaks it's a way of giving you the most important notes to be played to recreate the sound. If you add just a handful of these dominant frequencies, you'll get a signal which is almost but not quite the original signal, and if we do this process across many different little windows of our full signal. It's a way of telling us what notes should be played at what time and with what level of intensity, and if you do it right, you can get something that sounds like this, hey chopsticks repeat after me now i do have to come clean and admit that I am cheating a little bit by putting the words chopsticks is speaking up on the screen as humans. Eighty percent of the information, our brains gather about our surroundings, comes from our sense of sight, and so, as our dominant sense, our brain really leans.

On our eyes. Even when processing what we hear, if you don't believe me, here's two tests, you can do right now, number one go back and re-watch any part of this video where chopsticks is speaking only this time close your eyes and see how much harder it is to understand Him and number two, maybe you've, heard this audio clip from some random 80s toy, saying the word green needle now. Compare that to this other clip that says brainstorm only as you might have guessed. Those are the exact same audio clip which is bonkers, because those two words don't even have the same number of syllables for proof, i'll repeat it twice more only this time you can randomly pick a word to look at each time, alright, and so now, for the Big finale in a world's first exclusive, i present an actual piano attempting to play the world's hardest piano song rush e.

Of course. Buddy. Good luck. Do oh, the biggest problem with me, making a youtube video to try and teach you something.

Is that it's just not very interactive, which is why i'm pleased to say my friends at brilliant.org, who just so happen to be supporting this video have solved this issue and, if you don't know, brilliant is a problem-solving website and app. That has over 60 courses, including topics in math science and computer science, but their real special sauce is all about making the content interactive, which is way more effective at building your intuition. My favorite one so far is their scientific thinking course, where, instead of memorizing specific formulas that interactivity builds intuition about general principles, you see in the everyday world around you and it works great on both desktop and mobile. So, instead of picking up your phone and mindlessly scrolling, a news feed that leaves you feeling sad with brilliant, you get this addictive experience, which leaves you feeling stoked by unlocking the hidden principles that govern our amazing world.

So if you want to grow your brain and support my channel at the same time, go to brilliant.org markrover or use the link in the video description to sign up for free, because the first 2000 people get 20 off their annual membership. So thanks to brilliant for making learning fun thanks to chopsticks for not catching fire and burning my place down and, of course, thanks to you for watching.