The result is quite simply breathtaking. It looks like something shot for a movie using a stabilised dollycam, the fact they were able to achieve the same thing using nothing but a GoPro, their software, and likely a week of post-processing on a high end desktop PC is simply amazing.
I hope we see this technology actually become readily available. There might still be work to be done, but in general if they can reproduce the demo videos with other content then they're on to something people would want.
It appears that what they're doing here is simply extracting keyframes from the video, using them to compose a photosynth, then converting the autoplay of the synth to a video. If you load a photosynth and press "c", you can even see a the same point clouds and scene reconstruction seen on the research page.
It's striking but it's far more believable when you realize that they need to play at a much faster speed than the source, so they have tons of extra samples from which to extract information. They basically use all that data in the extra frames (that would otherwise simply get tossed away in a regular time-lapse) to construct a 3D scene. This wouldn't look nearly as good if they had to play it at normal speed.
It's certainly way better than the source video, but it's nothing close to what would come from a steadycam or dolly. You couldn't use this finished product in any kind of real production.
Yeah, it looks amazing. If the video is taken at faster speed (like 10X), then they can get a smooth realtime result when slowing it down at post processing.
Because they drop frames, they aren't stabilizing they are throwing away frames that move too much.
This is good stuff, I like it, but it isn't as wow as the structure from motion work.
And for the folks saying just up the framerate, that won't really help because the head motion needs to back in the same position as a previous frame. It is a function of how much and at what frequency the motion you want to remove is.
You think a week of post end processing? I didn't read the paper or anything so I could be way off base, but I would assume the algo simply has to choose which frames to keep and which to toss. I doubt this would take an enormous amount of time even with HD videos. The algo is most likely just really clever in how it chooses a good frame vs a bad one.
On the other hand, if it is actually generating a lot of "best guess" images to put between gaps that are too large to bridge (too many bad frames in a row) with the current frames I could see that taking a bit longer, but not a week.
Interesting that the final video ( mostly the rock climbing ) resembles a video game, where shapes and textures "pop-in" as they are rendered. The technical explanation video was really well done.
If the MSR researchers are here -- I'm curious what does it look like when bordering hyperlapse with regular input? i.e., if there were a video consisting of input frames at the beginning and the end, with a stretch of hyperlapse in the middle, what does the transition look like? Does it need much smoothing?
Also you probably saw this over the past week: http://jtsingh.com/index.php?route=information/information&i... (disregarding the politics of that) Whatever he's doing (I assume a lot of manual work) it has a very similar effect and it has these beautiful transitions between speeds.
This would be possible. Although it would require providing some UI so the user could specify which parts should be sped up.
I've seen the Pjong Yang video, it is beautiful work. It requires very careful planning and shooting, and a lot of manual work to create such nice results. We're trying to make this easier for CASUAL, but it's still FAR away from the quality of professional hyperlapse.
Hi spindritf -- I work for YouTube and have been looking into some mixed content issues with embeds. Mind if I email you and ask for some details about this scenario?
I walked around Boston once with some friends for 7 hours. When I remember it I see it as the hyperlapse, not moment for moment or sped up. Super interesting work.
Okay please sign me up. I'm willing to pay hundreds of dollars for this software. I have hundreds of gigabytes of time lapse that I've taken that is just sitting there because of lack of ability to do something. I'd easily pay $200+ for this software right now just so I can have those videos and free up massive hard drive space.
I'm not sure that this software is what you want. It takes regular speed and converts it to high speed. If you start with regular time lapse you'll need to speed it up even more - maybe as much as 10x. Now you'll have super fast time lapse.
I wonder how far you can get by using a "naive" timelapse of selecting frames from the video, but being smarter about which frames you choose. Rather than just choosing every nth frame, try to choose visually consistent frames by making the intervals between the frames loose, then apply conventional stabilization after the fact.
This was my initial thought about how they were doing this, but I don't think it's as applicable as it would seem. At 10x speed up, that's still ~3 frames from every second. I'd imagine a biker would spend at least a second turning to look down an intersecting road before continuing through. So that would be at least three frames where the perspective was heavily modified. It would have to select for right before and right after the head turn and ignore everything in between, which would probably create quite a jagged warp effect.
From the paper listed on the page it looks like it takes about 305 hours to process a 10 minute video. The vast majority of that is during the "source selection" phase which takes 1 minute per frame of video.
This is so insanely cool. I plan to get a GoPro some day soon and will take it on hikes in the Pacific NW. If I could turn my hikes into beautiful time-lapses like these, I'd be blown away.
I guess that it's not a proximity problem. For example, in the first video at 3:08 a gray mountain with snow appears on the upper right corner, and replaces a piece of sky. I think that a big rock was occluding the vision of the mountain in this frame, and the algorithm has to choose a texture from another frame to fill the void, and it made a mistake.
If you watch the technical video, they say that they couldn't use the scene reconstruction for the climbing video as there were too many artefacts. This is why the rendering isn't as good as the others.
Mind blowing results! Although the name Hyper-lapse doesn't really convey the goal, it should be named Smooth-Lapse, because thats what its doing. Too much hyper-x already.
I would use this for sure. I do timelapses of runs I do and set as challenges for our social running group. The source is a head-mounted GoPro.
The problem with them is that a straight forward pick-every-nth-frame gives a motion-sickness inducing video, as well as blurry. If you could extract the frame from the top of each stride when the camera is most steady, I would imagine it would be very much more watchable.
This is incredible. By watching the hyper lapse versions of the mountain climbing, you can clearly see which path is taken, are able to get glimpses of whatever paths are available. This would be a huge advantage for people learning how to rock climb. I can image that a similar situation would occur for many other activities. Great work!
This is great. I have weeks of footage from a camera that I wear around and would love to use that video to make a hyperlapse. I would also be interested in seeing how well this does with photos taken every few seconds as opposed to video. Although, after reading the paper, it looks like there would be a lot of optimization that would need to happen to make it more efficient. (Their original implementation took a few hours on a cluster.) Luckily, as they stated in the technical video, they haven't tried to do anything more than a proof of concept; so there is plenty of room to optimize. I'd be interested to see how well a single-machine OpenCL or CUDA implementation does compared to the CPU clusters they were using in the paper.
[+] [-] UnoriginalGuy|11 years ago|reply
I hope we see this technology actually become readily available. There might still be work to be done, but in general if they can reproduce the demo videos with other content then they're on to something people would want.
[+] [-] jjcm|11 years ago|reply
It appears that what they're doing here is simply extracting keyframes from the video, using them to compose a photosynth, then converting the autoplay of the synth to a video. If you load a photosynth and press "c", you can even see a the same point clouds and scene reconstruction seen on the research page.
Source: I worked on photosynth.
[+] [-] pron|11 years ago|reply
[+] [-] dublinben|11 years ago|reply
[+] [-] ww520|11 years ago|reply
[+] [-] sitkack|11 years ago|reply
This is good stuff, I like it, but it isn't as wow as the structure from motion work.
And for the folks saying just up the framerate, that won't really help because the head motion needs to back in the same position as a previous frame. It is a function of how much and at what frequency the motion you want to remove is.
This was on my todo list, item removed.
[+] [-] skizm|11 years ago|reply
On the other hand, if it is actually generating a lot of "best guess" images to put between gaps that are too large to bridge (too many bad frames in a row) with the current frames I could see that taking a bit longer, but not a week.
[+] [-] Mithaldu|11 years ago|reply
[+] [-] rkuykendall-com|11 years ago|reply
[+] [-] msane|11 years ago|reply
Also you probably saw this over the past week: http://jtsingh.com/index.php?route=information/information&i... (disregarding the politics of that) Whatever he's doing (I assume a lot of manual work) it has a very similar effect and it has these beautiful transitions between speeds.
Amazing work and the videos are stunning.
[+] [-] jkopf|11 years ago|reply
This would be possible. Although it would require providing some UI so the user could specify which parts should be sped up.
I've seen the Pjong Yang video, it is beautiful work. It requires very careful planning and shooting, and a lot of manual work to create such nice results. We're trying to make this easier for CASUAL, but it's still FAR away from the quality of professional hyperlapse.
[+] [-] jclarkcom|11 years ago|reply
[+] [-] spindritf|11 years ago|reply
https://www.youtube.com/watch?v=SOpwHaQnRSY
https://www.youtube.com/watch?v=sA4Za3Hv6ng
The hyperlapse of the climbing video looks like an FPS game from a decade ago with texture refreshing as you get closer.
[+] [-] jplevine|11 years ago|reply
[+] [-] jkopf|11 years ago|reply
[+] [-] jahmed|11 years ago|reply
[+] [-] ClassicFarris|11 years ago|reply
[+] [-] steven2012|11 years ago|reply
[+] [-] NeilSmithline|11 years ago|reply
[+] [-] moultano|11 years ago|reply
[+] [-] jdmichal|11 years ago|reply
[+] [-] iamshs|11 years ago|reply
I see they have listed a Windows app coming. Is that Windows desktop app?
[+] [-] runeks|11 years ago|reply
[+] [-] bane|11 years ago|reply
[+] [-] jkopf|11 years ago|reply
[+] [-] readerrrr|11 years ago|reply
[+] [-] sambodanis|11 years ago|reply
[+] [-] oliwarner|11 years ago|reply
[+] [-] adt2bt|11 years ago|reply
[+] [-] oh_sigh|11 years ago|reply
[+] [-] oniTony|11 years ago|reply
[+] [-] Lifescape|11 years ago|reply
Awesome idea/execution!
[+] [-] gus_massa|11 years ago|reply
[+] [-] ascorbic|11 years ago|reply
[+] [-] 0x0|11 years ago|reply
[+] [-] 31reasons|11 years ago|reply
[+] [-] tehwebguy|11 years ago|reply
https://www.youtube.com/watch?v=SOpwHaQnRSY&t=2m15s
[+] [-] itchmasterflex|11 years ago|reply
[+] [-] limsup|11 years ago|reply
[+] [-] photojosh|11 years ago|reply
[+] [-] washedup|11 years ago|reply
[+] [-] jelveh|11 years ago|reply
[+] [-] junto|11 years ago|reply
[+] [-] sabalaba|11 years ago|reply