Problem with optical flow on slowed down video

I've got some video of sea lions leaping above the water.  It was shot at 8K 29.97Hz.

The jumps are so fast that I would like to slow them down.

When I use normal the video isn't as smooth as I would like.

The seal lions are moving perpendicular to the line of sight to the camera.

However both frame blending and optical flow result in ghosting; a seal seemingly moving forward and then in the next frame back with the sea lion in the forward frame looking translucent.

Is this just the natural limitation of the blending and optical flow methods or is there some way to avoid it?

Thanks!

It all depends on how much you want to slow down the video. For fast moving subjects, it is better to shoot in 60p (59.94 fps) if your final project is going to be 30p (29.97 fps). This way, you can use "Automatic Speed" in FCP to achieve a very smooth natural 50% slow motion without needing any Optical Flow.

As you have shot in 29.97 fps, you could create a 23.98 fps project. Then drop the 29.97 clip on the 23.98 timeline, select the clip, and select Automatic Speed in the Clip Retiming options. This will already give you a slight natural slow motion. If that's not enough, you can further slow down the clip by 50%. If this produces stuttering artifacts, you can try Optical Flow to smooth out the slomo but there is no guarantee that this will work if the jumps are really, really fast. This is not a limitation of optical flow, it's a limitation of how your video was shot.


- Ronny 

trinko wrote:

...optical flow result in ghosting; a seal seemingly moving forward and then in the next frame back with the sea lion in the forward frame looking translucent....Is this just the natural limitation of the blending and optical flow methods or is there some way to avoid it?...

Optical flow must construct missing frames from the adjacent frames. It is what animators call "in betweening" or interpolation:  en.wikipedia.org/wiki/Inbetweening

Scene types involving complex backgrounds (esp. water) are difficult for various video processing algorithms, not just optical flow. That is why those scene types are often used in academic video processing research:  ultravideo.cs.tut.fi/#testsequences

There is an expensive 3rd-party optical flow plugin called Twixtor:   revisionfx.com/products/twixtor/fcp/

There is a new technique called DAIN (Depth-Aware video frame INterpolation) which promises to improve optical flow techniques.  It's not yet available in a commercial product but there is source code and a pre-compiled alpha version for Windows:  www.diyphotography.net/this-ai-software-...ally-zero-artifacts/

ronny courtens wrote:

It all depends on how much you want to slow down the video. For fast moving subjects, it is better to shoot in 60p (59.94 fps) if your final project is going to be 30p (29.97 fps). This way, you can use "Automatic Speed" in FCP to achieve a very smooth natural 50% slow motion without needing any Optical Flow.

As you have shot in 29.97 fps, you could create a 23.98 fps project. Then drop the 29.97 clip on the 23.98 timeline, select the clip, and select Automatic Speed in the Clip Retiming options. This will already give you a slight natural slow motion. If that's not enough, you can further slow down the clip by 50%. If this produces stuttering artifacts, you can try Optical Flow to smooth out the slomo but there is no guarantee that this will work if the jumps are really, really fast. This is not a limitation of optical flow, it's a limitation of how your video was shot.


- Ronny

Thanks! Unfortunately I had to shoot at 8K because the sea lions were so far away and then zoom in to produce a 1080p video that looked good.

Given that my camera, a Canon R5, can't shoot 8K at a higher rate that means that I'm stuck.

thanks! I read about Twixtor but since these videos are just for myself, my family, and friends it's a bit too expensive.

I was hoping that there was some way to tune the tweening to improve quality.

to be fair the final video using just normal to slow it down 10x isn't that bad.

vimeo.com/583207903

I have an R5, which I mainly use for advanced photo shoots. It's a very high-end and quite complex camera, I wouldn't really call it something for family and friends (-:

I agree that the slow motion @25% in your video does not look bad except for the fact that the picture is much too soft due to the heavy artificial cropping. At 10x slomo, I think it looks horrible with all that stuttering. 

I have downloaded your video, added the downloaded 29.97 video to a 23.98fps timeline, applied Automatic Speed and then set Retime to 25% with Optical Flow. The results are quite good, except in the middle of the shot where you move the camera abruptly to re-frame the sea lions. Optical Flow does not like abrupt camera movements in a shot, that's where you get the ghost frames. 

I also have Twixtor 7.5 Pro and I know quite well how to use it. I have tried to slow down your footage with Twixtor but it does not give any better results at all. So, the problem is not with your frame rate or optical flow, it's due to the heavy cropping and the abrupt movement in the middle of the shot.

The only way you can get great results with shots like these is by using appropriate gear during the shoot. If you want to shoot fast-moving subjects at a distance, use a fast telelens. And use a heavy tripod with a fluid head if you want smooth camera movements when shooting video. For these kinds of shots, I use a canon RF 100-500mm lens on my R5. With a lens like this, your footage is much, much clearer to start with and you don't need to crop artificially much or at all. Instead of having to shoot in 8K, you can use the 4K HQ @120fps setting in the R5 which records the best 4K quality video on the market today. Using 120fps, you won't have to slow down the footage artificially either. So everything will look much better out of the box. There's only so much you can fix in post, nothing beats recording video correctly from the start.

- Ronny

Well my wife bought it for me so it's for family and friends.

Historically I've been shooting mostly stills and the R5 is great for that.

I wasn't denigrating the camera, I love it, but just pointing out that since my videos don't bring in any money I can't really justify spending a fortune on plug ins.

As to the lens I used the Canon 800mm. The sea lions were over 200 yards away.

So while I agree that video quality isn't great changing lenses wouldn't have helped much. 

Without that abrupt camera sweep I wouldn't have captured the best leap, could I have moved that fast with a fluid head?, so I'm happy with it.

Which fluid head would you recommend?  Since I'm going to be collecting more video it'd be good to have the right equipment.

Also note that we have a 1080p TV and so on that the video looks pretty good at least to my untrained eye; for friends and family only.

Thanks for running the video through Twixtor.  I'll try and repeat your process with automatic speed etc and see how it works.  

I really appreciate your assistance.  Since getting the R5 I've been getting a lot more video and the 8K is giving me the magnification I need given the remoteness of my subjects. The 4K 119fps is also great for waves and jellyfish at the aquarium, not to mention spiders spinning their webs.

Since I just bought FCP about a month or so ago I'm still a newb so the help from the folks on this forum is very useful.