I assume it is looking for 5 fingers. I noticed if i folded my thumb in, it did not recognize the gesture so maybe an edge case where people missing fingers can't use this.
Also when I scratched my face briefly it did recognize it... funny.
I would strongly recommend trying to build a web demo somewhere down the line. Even if it requires flash to get the webcam working, I'm sure your conversions would go through the roof if you could show how well it worked instantly on your landing page.
Small note -- the blurring of the image is distracting rather than focusing. Down the line maybe think about a better landing page image.
We hated using the keyboard and mouse for controlling music in the background or watching movies on our computers from a few feet away. So we decided to solve this problem by using hand gestures to control music and videos via the built-in webcam in your computer.
Please download and try it. Do let us know what you think in your comments below or by emailing us at feedback [at] flutter.io.
I hardly ever get excited after downloading something, maybe a, wow that's cool, but when I installed this (which was very easy) and it just worked, I actually clapped and said 'this is fucking awesome.'
I had the exact same feeling. You see these gesture controls in movies and demos and they seem great, but it's truly exciting when you get to use it yourself.
I believe one of the co-founders is Navneet Dalal, who, along with Bill Triggs, developed the "Histograms of Oriented Gradients" (HOG) features that are widely used across all of computer vision†. Thus, I'm guessing that there's some serious vision in use for this app, which is pretty awesome!
(My apologies to the other co-founders, who are likely just as talented!)
However, music control isn't a pain point for me. What I would pay for is a start/pause gesture for Netflix/Hulu/Amazon Video. Worst thing about TV on a computer.
My friend and I looked into controlling those apps for a program we made to control media players with your voice called Shoevox - http://shoevox.org
The hard part was that we used sending keyboard shortcuts as our method of sending commands (which works for programs we support like WMP or VLC), but we couldn't do that reliably to Flash. We thought about switching to UI Automation but haven't done so yet.
I just ask my very non-technical girlfriend to come see this new app I downloaded. I showed her flutter.io working in action. She shrieked and said, "that's awesome, how do I get it?". That's a very good sign for you guys...nice job.
Though I know there's a chance you'd not be willing to share the information, I'd love to learn a little bit about how you built this product.
It appears as though you're using ML to recognize the gestures—anything else you could share would be fascinating. I'll be sending in a job application, as well!
You're right. We use image processing, computer vision & machine learning to detect Flutters. Our entire tech stack is custom written as we are solving some quite challenging & difficult AI problems.
Interesting/cool project, though I do wonder how you plan on keeping it simple when you expand and add additional actions?
The UX of controlling media using the Xbox Kinect leaves a lot to be desired - it's cumbersome.
I've had my mac mini connected to my tv for awhile and am very interested in the Internet TV space in re: what will become the standard way we control such technology? Will it be thru gestures, airplay on iphone/ipad type technology, voice driven or a mix of all?
For many years I've been using a wireless gyration mouse/keyboard (demo of Internet TV setup http://www.youtube.com/watch?v=l-IryK4wsUM) and recently I started to use Chrome's voice search to quickly bring things up especially if my wireless keyboard isn't reachable.
After seeing this and my years of enjoying the Internet on my TV I'm thinking a few short gestures like this app uses and mix of voice driven actions might provide the best UX.
Based on the views and comments of video I link to above people want the Internet on their TV and all it provides. We just haven't figured out the best UX for it.
Paul - that is a great observation and feedback. We believe that future will be about us using touch, voice, and gesture all three depending on context and circumstances. We do plan to keep it simple, and we have some great ideas for it.
"The music swirled and dived for a moment. Another voice broke in, presumably Halfrunt. He said "Vell, Zapbod's just zis guy, you know?" but got no further because an electric pencil flew across the cabin and through the radio's on/off-sensitive airspace. Zaphod turned and glared at Trillian -- she had thrown the pencil. "
This is very cool. It worked perfectly even in the dimly lit room I am sitting in.
What I've been wanting and have thought about developing myself is a universal play/pause control that pauses whatever you happen to be listening to at the moment, whether it be a desktop client like iTunes, Spotify, Rdio, etc, or if it is something running in a web browser like Pandora, Soundcloud, Youtube, etc.
This app takes a step towards this in the way it handles iTunes and Spotify at the same time, by controlling whichever app had focus last.
I imagine for web browser control a browser extension would necessary. And ideally a plugin architecture would exist to allow 3rd parties to add support for additional apps.
A different but functionally similar idea would be an automatic pausing of all other services when you start playing one (i.e. when I start playing a Youtube video, automatically pause iTunes).
Really cool. I was just playing with it and my girlfriend walked over to ask me a question. I held up my hand to my computer as if to say "Hold on, computer."
It was a bit surreal. Very cool. Can't wait for more gestures or a generalized way to program my own gestures. I would love to be able to gesture "Switch to iTerm.app" by doing an OK sign or something. Sign language letters maybe?
Anyway, thanks. Really enjoy it already, can't wait for more. Also, I would pay money for this. If it had all of the play/pause/forward/back functionality I think I'd give you 4.99 if it was accurate. Just sayin'.
My thoughts almost exactly on the programmable front... like Better Touch Tool but rather No touch Tool, although the latter name does retain the locker-room hilarity of the former.
Perhaps a gesture to bring up the app switching ribbon and a gesture to reference the item (numbered from left to right maybe.)
I'd also like to have the option to calibrate it, my eyes are painfully photosensitive and I tend to work in very dimly-lit conditions.
What's nice about this is you don't need to switch screens to pause the music if you find yourself writing/coding and need some quiet when you're in the zone. Granted you can do this with the mac play/pause physical buttons but that would require your eyes to look down and pause your thought process.
This happened to me just now after I installed Flutter, and it seemed to me that physical gestures didn't interrupt my higher brain functions. Of course when I noticed this it made me interrupt my work and come to HN to leave this comment. The irony.
Glad you noticed it. We want you to never think about controlling your music at all. This is one of the things I love too... we listen to music in a background, so controlling it should be in the background too.
That's what I like most about it too. Love not having to switch screens to stop the music when a call comes in. One thing: I didn't find the installation super intuitive. Probably some room to improve that experience
By the way I see great application in this for education, particularly with smaller kids
Flutter seems to keep the camera on 100% of the time which prevents apps like Skype from using the camera for video chats. I'm not sure if there is a good workaround for that?
Andrew - great observation. On top of your menubar, there's Flutter icon. Click on it, and it will give you simple flutter on and flutter off capability.
There is another, more fully-featured app called FluidTunes which I used quite a while ago. Not sure if it still works on the latest version of iTunes: http://majicjungle.com/fluidtunes.html
This is awesome, but one question/suggestion: when a user is going through the "How to Use Flutter" tutorial, do the gestures work then, or only once the tutorial is closed? I kept trying to do it, but there wasn't a response until I closed the tutorial.
Only when tutorial is done and you're using iTunes or Spotify. First time you bring up either of the apps, it will notify you through top menu bar that Flutter is controlling it now.
Once that happens, you can control in the background or even if it is minimized.
Few questions: Aren't you entering a field which has patents covering gesture controls on media exactly like this?
A comparative advantage of mice and keyboards is that a minor amount of energy, (but large amounts of co-ordination) are leveraged into larger changes.
Wont this limit the number of the gesture controls you can incorporate?
(OTOH - if you had enough fidelity, you would just need a dummy keyboard. You would just have to map finger movements with key presses. you could just create a webcam of your own as well.)
Very cool although with CPU usage at about 30% on my Air I won't be able to use it much. If you can figure out a way to drop average CPU usage down to reduce heat & power usage that would be a big deal.
Hi Hedgehog - thats a great point. We thought of it, and when you're not doing a gesture, CPU usage should drop to around 2-3%. Let us know if it doesn't. In our test, average CPU usage was around 5%.
[+] [-] tsunamifury|14 years ago|reply
I assume it is looking for 5 fingers. I noticed if i folded my thumb in, it did not recognize the gesture so maybe an edge case where people missing fingers can't use this.
Also when I scratched my face briefly it did recognize it... funny.
I would strongly recommend trying to build a web demo somewhere down the line. Even if it requires flash to get the webcam working, I'm sure your conversions would go through the roof if you could show how well it worked instantly on your landing page.
Small note -- the blurring of the image is distracting rather than focusing. Down the line maybe think about a better landing page image.
[+] [-] shearn89|14 years ago|reply
[+] [-] mehuln|14 years ago|reply
Please download and try it. Do let us know what you think in your comments below or by emailing us at feedback [at] flutter.io.
Currently, it works for Mac OSX 10.6 and up.
[+] [-] wmw|14 years ago|reply
[+] [-] joshmlewis|14 years ago|reply
[+] [-] nikcub|14 years ago|reply
which kept switching your music off and on?
[+] [-] ajju|14 years ago|reply
Love it!
[+] [-] mehuln|14 years ago|reply
[+] [-] apu|14 years ago|reply
(My apologies to the other co-founders, who are likely just as talented!)
† http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients
[+] [-] mehuln|14 years ago|reply
[+] [-] lukev|14 years ago|reply
However, music control isn't a pain point for me. What I would pay for is a start/pause gesture for Netflix/Hulu/Amazon Video. Worst thing about TV on a computer.
[+] [-] mehuln|14 years ago|reply
[+] [-] rsbaskin|14 years ago|reply
The hard part was that we used sending keyboard shortcuts as our method of sending commands (which works for programs we support like WMP or VLC), but we couldn't do that reliably to Flash. We thought about switching to UI Automation but haven't done so yet.
[+] [-] paul9290|14 years ago|reply
[+] [-] twakefield|14 years ago|reply
[+] [-] mehuln|14 years ago|reply
[+] [-] ihodes|14 years ago|reply
It appears as though you're using ML to recognize the gestures—anything else you could share would be fascinating. I'll be sending in a job application, as well!
[+] [-] mehuln|14 years ago|reply
[+] [-] toomuchtodo|14 years ago|reply
[+] [-] paul9290|14 years ago|reply
The UX of controlling media using the Xbox Kinect leaves a lot to be desired - it's cumbersome.
I've had my mac mini connected to my tv for awhile and am very interested in the Internet TV space in re: what will become the standard way we control such technology? Will it be thru gestures, airplay on iphone/ipad type technology, voice driven or a mix of all?
For many years I've been using a wireless gyration mouse/keyboard (demo of Internet TV setup http://www.youtube.com/watch?v=l-IryK4wsUM) and recently I started to use Chrome's voice search to quickly bring things up especially if my wireless keyboard isn't reachable.
After seeing this and my years of enjoying the Internet on my TV I'm thinking a few short gestures like this app uses and mix of voice driven actions might provide the best UX.
Based on the views and comments of video I link to above people want the Internet on their TV and all it provides. We just haven't figured out the best UX for it.
[+] [-] mehuln|14 years ago|reply
[+] [-] Roritharr|14 years ago|reply
Implement it for Android and you can charge me 30€ so i can easily control the mp3 player in my car.
[+] [-] joshmlewis|14 years ago|reply
[+] [-] mikeknoop|14 years ago|reply
[+] [-] bdunbar|14 years ago|reply
HHGTTG - Douglas Adams
[+] [-] plasma|14 years ago|reply
[+] [-] cellularmitosis|14 years ago|reply
Expected reaction, having a video demonstration of the product on your landing page: "THIS IS FRICKIN' AWESOME!".
Please, please, but a video on your landing page. You will thank yourself after checking your conversion rate.
[+] [-] mehuln|14 years ago|reply
[+] [-] nileshk|14 years ago|reply
What I've been wanting and have thought about developing myself is a universal play/pause control that pauses whatever you happen to be listening to at the moment, whether it be a desktop client like iTunes, Spotify, Rdio, etc, or if it is something running in a web browser like Pandora, Soundcloud, Youtube, etc.
This app takes a step towards this in the way it handles iTunes and Spotify at the same time, by controlling whichever app had focus last.
I imagine for web browser control a browser extension would necessary. And ideally a plugin architecture would exist to allow 3rd parties to add support for additional apps.
[+] [-] goronbjorn|14 years ago|reply
[+] [-] devin|14 years ago|reply
It was a bit surreal. Very cool. Can't wait for more gestures or a generalized way to program my own gestures. I would love to be able to gesture "Switch to iTerm.app" by doing an OK sign or something. Sign language letters maybe?
Anyway, thanks. Really enjoy it already, can't wait for more. Also, I would pay money for this. If it had all of the play/pause/forward/back functionality I think I'd give you 4.99 if it was accurate. Just sayin'.
[+] [-] mark_integerdsv|14 years ago|reply
Perhaps a gesture to bring up the app switching ribbon and a gesture to reference the item (numbered from left to right maybe.)
I'd also like to have the option to calibrate it, my eyes are painfully photosensitive and I tend to work in very dimly-lit conditions.
[+] [-] intenex|14 years ago|reply
[+] [-] siavosh|14 years ago|reply
This happened to me just now after I installed Flutter, and it seemed to me that physical gestures didn't interrupt my higher brain functions. Of course when I noticed this it made me interrupt my work and come to HN to leave this comment. The irony.
[+] [-] mehuln|14 years ago|reply
[+] [-] esharef|14 years ago|reply
By the way I see great application in this for education, particularly with smaller kids
[+] [-] joejohnson|14 years ago|reply
[+] [-] andrewpi|14 years ago|reply
[+] [-] mehuln|14 years ago|reply
[+] [-] escalona|14 years ago|reply
[+] [-] cmelbye|14 years ago|reply
[+] [-] basil|14 years ago|reply
[+] [-] mdanger|14 years ago|reply
[+] [-] mehuln|14 years ago|reply
Once that happens, you can control in the background or even if it is minimized.
[+] [-] intended|14 years ago|reply
Few questions: Aren't you entering a field which has patents covering gesture controls on media exactly like this?
A comparative advantage of mice and keyboards is that a minor amount of energy, (but large amounts of co-ordination) are leveraged into larger changes. Wont this limit the number of the gesture controls you can incorporate?
(OTOH - if you had enough fidelity, you would just need a dummy keyboard. You would just have to map finger movements with key presses. you could just create a webcam of your own as well.)
[+] [-] hedgehog|14 years ago|reply
[+] [-] mehuln|14 years ago|reply