Masking complex objects

Hi all,

over the last couple of days I have been trying all sorts of ways to get a working method to mask (and track) complex objects, such as people or just heads. One of the things I would like to do is creating a depth-of-field effect around the heads. I know there are plug-ins available that might do an acceptable job, but they come at a price...
So I applied the Gaussian blur to a clip and then tried to mask out areas that should not be affected. The first attempt was to draw a mask, but had to abandon that idea because of e=the enormous amount of work to track the mask throughout the clip. So I tried to combine color masks and shape masks, but ran into a number of limitations in FCPX (unless I am completely wrong, which I hope I am...)
First, it does not seem to be possible to combine multiple color masks in the same effect. It is only possible to use a single color mask and add multiple shape masks. That brings me to the second limitation: I can invert the masks, but only all at the same time. I have not found a way to invert individual masks.

What am I missing ?

Manually key framing your masks can be quite extensive depending on the clip and amount of movement it contains, as you have found out. Applying a color mask from the built in color wheels or color board of the inspector does indeed limit you to a single selection for that particular control board, but you can add multiple corrections to get another instance of an independent color mask. For example, you can have 5 or more instances of the color wheels (or color board) on a clip and each one can contain a single color mask.
The shape masks, when added similar to above cannot be individually inverted or non-inverted, also as you have found out. But, exactly like the above example for the color mask, you can add them separately with additional instances of a correction. All the shape masks in any given instance will invert together. But another instance can allow you to add one that is not inverted.
Another option is to add a shape mask (or as many as you need) from the masks section of the effects browser. That way, every mask is entirely independent and can be made to invert or not.
I would highly recommend looking into TrackX from Coremelt for serious masking/tracking projects as it will do all the heavy lifting for you.

… method to mask (and track) complex objects, such as people or just heads…

In post? Close to impossible, when thinking of hairs, motion-blur etc.

On iOS/iPhoneX, the camera records the z-positiion = depth of subjects and stores this info in its special file format. Then it's possible to apply blur&bokeh in post.

With regular cameras, your only option, imho, is, to record two shots: a clean-plate of the scene = without person, and a second shot, with person and a tiny greenscreen; in post, you can mask the person with a color-key, and replace the bg with the other shot.

… things get ugly, when you want to move/zoom camera LOL…

Tools like TrackX help tracking 'areas' but lack detail (blonde hair in front of white bg for example) …

No, as long as Apple doesn't add its depth-recording into FCPX ("key anything behind this point!"), for a detailed mask color-key (on recording) or painstaking rotoscoping are your only options…

Thanks Karsten. Yes, ideally you plan a shot like that, but unfortunately that is not alway possible. Adding the effect in post, as you say, proved to be impossible.

Dinky wrote: … Adding the effect in post, as you say, proved to be impossible.

uhm, well, not impossible but … painstaking.
All those Hollywood blockbusters which get a 3D-finish, although recorded in 2D proof me wrong.-
But 'masking' manually = rotoscope is just PITA…

So the invention of compo, luma-key, soda-vapor keying, wotever … and, latest, z-data-recording

Interesting trivia:
iOS creates 'depth info' with just one camera on the iPhoneXr…pure computational by identifying content ("faces are nearer camera than backgrounds") … nothing wrong to implement that in FCPX10.5, Apple, the 'Vision Framework' in Mojave has all needed components!!! :silly:

It goes without saying that for a task like this, there is much room for improvement, even if you rotoscope and track in a more advanced software for this like AAE - where you have a more powerful version of Mocha and also roto brush (similar to the magic wand selection tool in PS, only for video). That's where AI/deep learning should come in.

I once did a fake sDoF for a whole 5 minute video, in Motion. Looked very surreal, couldn't have achieved anything like that with my (then) 35mm-adaptor. Took a week and made by back hurt. Learned the hard way that one better splits complex moving outlines into many masks that you (manually) track and keyframe-animate individually. Nevertheless, an ordeal.

Yes, I tried multiple masks as well. Although it is easier than one big mask, I found myself asking why I was doing it, because the result is bad anyway. Especially the hair outline will never work

Thank you all for your thoughts about this

I would like to get back on the multiple color masks issue. I know I can add multiple color corrections each having their own color mask. In my case, though, I would like to apply multiple color masks to the same effect, like the Blur effect. I can add multiple shape masks, but only a single color mask, unless I missed something completely...

A DoF effect could be easier applied, if I could just add the Blur effect and mask it with multiple color masks. E.g. one for the hair, one for the skin, etc. That way I wouldn't need to keyframe anything.

Is there a way to achieve that ?

@Dinky:
I wouldn't sum Blur under ColorCorrection but an 'effect'… And there is a fine grey line btw. (Color)Keying and Masking … keys traditionally use a luma or color value to select an area, whereas masks select an area 'geographically' << hope you understand what I mean :blink:

But…
Just reading Gruber's test of the iPhoneXS and stumble upon this:

… Any iPhone or iPad running iOS 12 can edit the bokeh depth of field of a Portrait Mode shot from an iPhone XS or XR. Even better, when Mojave ships, Photos for Mac will be able to do it too. …

… which let me assume, that feature will make it into FCPX v10_5.

AND KEEPS THE POINT UNANSWERED HOW AN iPXr computes 'depth' with a single camera - unless, it's not real z-data but 'subject recognition'! (…" this is a face=near, this is background= far/blur…") - with a precision of single strands!

hmm, probably just me finding this 'interesting', but wait&see I guess …

The reason I mentioned color corrections is, that it is possible there to add multiple corrections, each having its own color mask. In e.g. a Blur effect (used to simulate DoF) I can only add a single color mask (and multiple shape masks). Adding multiple instances of the Blur effect does not work. So, if only masks in effects could also consist of multiple color masks that could be inverted individually, it would make life a lot easier.
Would be nice if FCPX 10.5 would help here

Dinky wrote: So, if only masks in effects could also consist of multiple color masks that could be inverted individually, it would make life a lot easier.
Would be nice if FCPX 10.5 would help here

We had this discussion, and it’s unlikely to change in the foreseeable future. The reason is that FCP’s order of operations is “destructive”, meaning that the next filter takes the output of the previous one . You had to start with a fresh layer copy of the source (that’s of course possible with garbage masks asf.), but this would be easier to handle in a node-based app. Free ones are Blender (unlimited power, but learning curve is as steep and tall as Mount Everest) and Fusion 8.
But anyone who is familiar with Photoshop probably is not too uncomfortable with the said layer clone concept. It’s just that FCP X’s GUI doesn’t help much to keep track of all masks and layers and how they relate to each other.
Motion does. Motion can also combine individually keyed colors with garbage masks. I’d try this in Motion if I were you.