Workflow for Transcoding Media Outside FCPX

Greetings everyone! Long time reading first time poster. Anyhow...

So I'm coming up to speed with FCPX. I've gone through Lynda.com's training and feel like I have a good handle on the basics. It's about that time to start a production project and I'm working with another cameraman/editor as well, so we're nailing down our workflow.

The way we used to do it (FCP7):
1. Ingest all our H264 footage from card to harddrives.
2. Dump it all into Compressor, transcode to ProRes422 LT.
3. Dump all that footage onto a fast drive and subsequently into FCP7.
4. Edit

However, what I'd like to do now is a bit different, mainly because we're also going to be cutting on lower powered notebooks that would greatly benefit from proxy media. So here's what I have so far....

New way (FCPX) that I haven't worked out:
1. Ingest footage from cards to harddrives (ok no big deal there).
2. Transcode with Compressor (we're also looking into Sorenson Squeeze, but not a big deal)
3. Import? Wait, how do we tell FCPX to import the 422 but use the Proxy?
4. Post on FCP.co....

So the question really, is how do I get FCPX to work with my 422/Proxy media correctly when we're transcoding it with an external application? Is it as simple as dragging the proxy files into the "Proxy Media" directory? Also, I've discovered that if you import ProRes 422 media and tell FCPX to copy it to it's folder, it puts it into Original Media, not High Quality Media. Which I guess could be argued as a sensical action. But I digress.

Any help is appreciated!

After a bit of tinkering I've discovered that if I drag the pre-transcoded proxy media into the Proxy Media folder FCPX only recognizes it as online after I restart the app. Is there a way to force FCPX to try to reconnect media?

Also, just foolin' around I was noticing that regardless of whether or not I have FCPX do the transcoding, or if I do it externally, when I drag my clips onto the timeline, even for the first time with no transitions of any kind, it shows as orange and requires rendering. What's it rendering exactly? And if it's transcoded to ProRes 422 why is it rendering anything to just place a clip onto the timeline?

Are you transcoding to prores proxy and 422 or just proxy then reconnecting to the h.264 files for output??

No, Import the H.264 files DIRECTLY into FCP X, and there's no need for proxies, period, done.

You should also have been using the Log & Transfer function in FCP 7 with the plugin supplied by the DLSR manufacture.

@Mikey - I am transcoding the media into ProRes 422 and ProRes Proxy. So both ending up with two sets of files.

@BenB - I'm trying to avoid directly importing the H264 files into FCPX because the notebooks aren't really up to scratch to transcode the media. I mean, they can certainly do it (they're both MacBook Pros circa the 2.4 - 2.6Ghz Core 2 Duo years) but I don't see how trancoding the project on each system (at this point we're dealing with 2 notebooks and two workstations) is a viable solution to anything really. Considering our last project had roughly 20+ hours of footage shot over 2 days, we're looking to trancode on our fast workstation then disseminate it to all the notebooks.



EDIT: I should point out that while I'm impressed with FCPX's ability to transcode while you edit with the H264 originals, on a slower system all it really means is that everything moves at about 60% for the HOURS it takes to transcode a project's worth of footage. I appreciate that on a Mac Pro with beaucoup specs it happens seamlessly, but in the real world I'd rather set a trancode to happen overnight when it affects no one, then edit with the trancoded files at full speed.

If you edit H.264 originals, there's NO NEED TO TRANSCODE AT ALL!

I have edited H.264 on older MBP's with no problem. FCP X handles it just fine. No need to transcode at all. And why on earth are you transcoding to both ProRes and Proxy? There's no need for Proxy media when using DSLR originals at all. Use FCP X on your MPB with the native H.264 media. I know someone doing it with an Air and it runs just fine.

You're workflow makes no sense, and actually creates more overhead than necessary, even your FCP 7 workflow.

Yes, those laptops can edit the H.264 natively just fine, no Optimizing, no Proxy. And why proxy at all? What is the purpose of that?

@zeroscenecred I've noticed that FCPX does work best with non Long GOP formats, Even editing HDV (Which worked OK in FCP7) Still has issues. So transcoding most media is best, but I agree if you could do the transcode outside with compressor or another program that would be best. Have you looked at grinder from magic bullet, its a very simple program that does just that transcode media to a format for editing. I've used it in the field and its just as fast as compressor or you can try batch transcoding from mpeg stream clip and it works pretty well, the only problem with Mpeg Streamclip is it always forces the meta data to had a dominate field, and you would have to batch change the clips in FCPX back to none. I am going to try this on my next project.

@BenB the Log & Transfer function in FCP 7 only worked with some camera models, such as P2, h.264 from Canon, and some AVCHD cameras, Long and transfer was also slower than compressor because it did not use all your computers processors (A limit of the 32bit FCP7), the only benefit you tended to get with L&T was the ability to keep certain meta data, and timecode, but it was just as easy to add AUX TC to any clip. H.264 in FCPX works ok but on long format work anything over 30 min, it really starts to bog down, also if you use optical flow with long GOP formats you find that it creates some weird issues, but if you had transcoded the same clip do the same speed change and use optical flow on a ProRes file it does not create the same artifacts.

Actually, Log & Transfer is well documented to be much faster than Compressor, because the camera manufacturers created their own algorithms, and sped things up quite a bit. So what you're telling me defies my own studio's experience, and the experiences of my students and consulting clients.

L&T was much faster than Compressor, and all Panasonic, Sony, and Canon cameras worked with it, as those manufactures created their own plugins for it. We are currently waiting for these camera companies to re-write these plugins for FCP X.

What you're saying about Compressor being faster than FCP 7's L&T simply is not true. It theoretically makes sense, as long as you don't take into account that some formats are not multi-thread, so even Compressor couldn't use multiple CPUs. L&T actually did use multiple CPU's for the formats that allowed it.

Again, not sure why you made ProRes and Proxy media at the same time, if you're coming from a DSLR camera.

FCP X is great with long form projects, we've done two feature length projects so far and love it. Be sure you have enough RAM, powerful enough graphics card, and fast enough hard drives, and understand proper workflow.

Why are you making proxie files if you're making ProRes files?

@Mikey - Transcoding does seem like the way forward. Gets everything nice and "square" if you will, before you start. Magic Bullet Grinder looks pretty good. I looked at it once but don't remember why I decided against it. I might look at it again actually because I've heard it does a bang-up job. Compressor's been okay but it's kind of a weird app to use. Wish it had more options for where to intelligently put the output media and how to name it. Sorenson Squeeze has, for the time being, been removed from my short list despite seemingly having the best featureset and ease of use, because we're getting a gamma shift (brighter) on all our footage that gets H264-to-ProRes'ed. Not sure wtf that's all about but it's totally unacceptable. I'll look at Grinder tonight and see what I can find though!



@BenB - We never edit with H264. I don't know enough about the particulars of the format, or even have time enough to care, but every editor I've worked with past and present, handles the majority of their editing in ProRes, and so do we. I've noticed that FCPX slows down once you start stacking cuts and really working over your footage, when using H264, so we just go right to ProRes and don't have a problem.

We're creating the 422 full res media for the purposes of color correction, slow motion, motion graphics during/after editing, and so nothing is lost in the mix. The proxy media is used for easier playback on our notebooks. Playing back 1/2 res proxy clips is considerably easier on our notebooks than the full 1080p, and FCPX seems to handle proxy/fullres conforming seemlessly so there's no need to do lengthy manual conforms. In the end the proxy clips generally have an even SMALLER file size than the H264 clips (or just a tad bigger if memory serves me).

Lastly, regarding L&T. We've used L&T for one of our cams (a panasonic), but we're using Nikon HDSLR's in addition to that camera and they're not supported in L&T, nor does Nikon appear too concerned about releasing plugins. Ultimately though, it doesn't make sense for our small shop to handle two different workflows based on camera (panasonic through L&T, nikon through something else) because it leaves too much room for error, something getting lost, and doens't take into account a lot of other factors in our media management workflow (namely things like file naming, triplicate backups, injecting timecode into the Nikon footage, etcetera etcetera).

So while I appreciate that you're promoting and suggesting a simple FCPX workflow of "dump your H264 in and go" that doesn't work for us at all. I could (and have) written a paper on our own workflow for our internal use, and would be happy to share our humble findings and preferences in greater detail if anyone's interested or if you're truly curious why we're going through intermediate steps to ingest our footage. We aren't experts but so far it seems to work very well for us.

We are editing Canon H.264 native on a MacBook Pros with no problems at all. I know someone using an Air and loves it. I'd love to see that paper. Based on my many, many years of consulting for broadcast stations, sounds like something isn't being done correctly. I'm just saying...

Listen I get it. You've got years of experience that far outweigh mine. In fact, I yield to your superior experience, and make no claims that what I've written is anything even remotely authoritative or groundbreaking. In fact it's just a list of steps we go through for our workflow based on what works for us, so that when we bring in more people to our company we can easily say "this is how we work". Hell a bunch of the ideas are adapted from famous working professionals! Anyhow, I'll upload it onto our webserver and link to it.

That said, you're probably one of only a handful of people who I've ever seen advocate editing with H264, a highly compressed capture format, versus ProRes which is, by the remaining accounts, the industry standard. I'm just saying.


Since you don't seem inclined to answer my "can I force FCPX to reattach media" question or any others, let me go ahead and upload that paper because, all this macho d*ckwaving aside, it might actually benefit someone to read about what we do. And frankly I'd love to hear anyone's feedback on our system because we're always looking to make it better or faster. Paper should be posted by tomorrow. First I need to eat some dinner, have a beer, and generally do something more enjoyable than all this on my Friday night.

I've only stated my experiences with the systems, and things I know to be facts. I don't need the insults. I've tried to help, but you never answered the troubleshooting questions I posted. Best of luck.

LOL. Whatever floats your boat, chief. I've rarely encountered someone as condescending or dismissive especially in any professional community, responding to someone's literal first posts with no rational argument to even support your claims besides "trust me I'm an authority (paraphrasing)" and "period. done." Not to mention rude and shouty with your ALL CAPS. So yeah excuse me while I remain a bit skeptical.

But hey, no matter. I don't think you left any CD's at my house so this should be an easy and clean breakup. There's a big beautiful world out there full of people who no doubt know far more than both of us combined, who aren't you, who I will actually have a pleasant time dealing with. Cheers.

Speaking of....

@Mikey - So I downloaded Grinder and, to my pleasant surprise, I apparently hadn't dug deeply enough into the settings to find that it handles pretty much whatever I need it to do. So I set about running a test to see how it would work with X and that's when the wheels fell off.

I ran a very short clip for testing through Grinder. I setup the straight forward settings to what I needed: for the main transcode output ProRes 422 at full quality and size, and for the proxy output 1/2 HD ProRes Proxy format (which is what X produces when you tell it to create optimized media and proxy media). The resulting files looked and acted fine in Quicktime. I then dropped them into the Event's appropriate folders (high quality media and proxy media, respectively) only to find that X was showing some bizarre icon and no preview of the clips. What the heck? So I restarted X. No dice. Then I deleted the files (leaving only the original 264 clip) and suddenly it worked again. I tried it on a second system and again, same thing. Clearly X doesn't like something about those Grinder files. Really bizarre!

So I dunno what's up with Grinder and X but they don't play nice, at least not on my system. Not sure what's going on so I'm going to run a few more tests and see what I can see. I'll let you know how it goes!

Regards.
Alec

That's my workflow, too... Mbpro late 2008...

For what it is worth. I have a very long in the tooth 2007 MacBook Pro that I use for editing on the road, doesn't even have a qualified graphics card for FCP X. It can't handle h264 without dropping frames.

I apologize if I missed something in your post, but It wasn't completely clear if you had tried using the built in FCP X proxy workflow. I'm using it right now on a long format HDV project and it works like a charm. After importing the HDV footage, I started editing and realized the MacBook Pro was not up to the task. So I just selected all my clips in the browser, right clicked and chose Transcode, and then Proxy. It made the proxy files in the background and then I just switched over in preferences to play Proxy Media.

Now I'm editing and even color correcting at a decent speed on my old MacBook Pro.

The "export media" will still use the Proxy Media if it is set in preferences, so you can use that for your draft proofs. When you're ready for final output, just switch preferences to "High Quality" or "Better Performance" and you'll get a high quality export.

I apologize if I am stating the obvious.

@tpayton - That's totally what I'm doing too. The only thing I was trying to find out is if FCPX has to be the app that creates the transcoded media, or if you can trick it into playing nice with media that you transcode into ProRes outside of FCPX. After playing with things for a few hours this afternoon and yesterday, it appears that, as best I can tell, you need let FCPX do the actual transcoding process if you want it to work right. But you're right, the settings to allow you to work in proxy mode totally allow you to edit on a less than qualified MBP and is the way to go.

For those that care, yes I was just trying to do all my transcoding outside of X so it could be run overnight. I've found the following....

1. If you transcode to 422 full quality and use those files as original media (i.e. you drag your ProRes'ed 422 footage into the Event) the clips will be fine and it won't ask you (or allow you to) generate optimized media because it's already optimized (duh!).

2. If you then try to "trick" X into thinking it's transcoded Proxy media by copying Proxy files you generated in, say, Compressor or Grinder, into the "proxy media" folder in Finder, you're sh*t outta luck and X will show you a weird icon. This is the case even if you make your ProRes Proxy media using the exact same settings X would have used. Weird.

3. However, if you let X generate the proxy media it seems to transcode it about 2x as fast as the external apps (i.e. Compressor, Grinder, etc) which is at least a nice thing.

4. I haven't yet tried to do the experiment from #2 above but using X-generated Proxy media (e.g. have my Mac Pro generate the proxy media then copying those files to my MBP, then moving those files into the Proxy Media folder in Finder).

5. My guess is that X is either doing some sort of checksum to make sure the proxy media matches the original footage, and storing that data somewhere. I'll check this later by modifying a proxy clip in another app and see if it breaks it back in X. This would be useful to know if you're trying to hack some sort of roundtrip system into your workflow.

Is any of this making sense to anyone or am I just taking crazy pills?




God I loved the movie Zoolander.

I was just trying to do all my transcoding outside of X so it could be run overnight.

I do my FCP X transcoding overnight. Just set it and leave. You can tell it to transcode footage from many different events at the same time. I do this and monitor things with my iPhone and LogMeIn.

My guess is that X is either doing some sort of checksum to make sure the proxy media matches the original footage, and storing that data somewhere.

Sorta. Actually what they do is create a unique ID for each clip in an event. (export a FCPXML to see this) Then when it creates proxy or high rez imagery it creates that containing the ID. Therefore they are all linked up and interchanagle via FCP X.

This is actually very cool, because you can actually remove the original media clips from the FCP event (at the finder level) and you can still edit everything because it will pull the proxy media because of the IDs. You can then kinda invent your own offline workflow there by removing the original files, and then copying the event to an external drive for remote editing. After remote editing bring back the event and put the original files back to the even (you'll want to merge the folder incase you added any more original media files during your remote editing).

BTW. These IDs are the reason you can't currently reconnect media with those of the same name.

The files "CurrentVersion.finalcutpro[event/project] are actually databases based on Final Cut Server functionality. FCP X is "database" centered. Projects link to Events, not individual clips. Only Events have any data about where individual clips are located. Here's something I use in my seminars for first time FCP X users.

fcproxuniversity.com/video/FCPXMMpt1.mov

This is all very good info, thank you tpayton and benb!

I think my only aversion to letting my MBP run overnight to generate proxies is just that our projects (well, at least the one we're currently working on) generated about 24 hours of footage from a two day shoot, and even letting my MBP create the proxy clips for all that would take at least 24 more hours (it seems like they do them at less than 1x though, oof).

What I'm hoping is that we can cook up a style of offline workflow, like you said tpayton. We find ourselves on planes pretty frequently and wanting to do rough cuts with the hardware we have which is more CPU limited than storage. But you get the idea.

Quick question: I know FCPX puts the "Final Cut Events" and "Final Cut Projects" into your ~/Movies folder by default, but can each external hard drive be setup to have those folders as well? We could, conceivably, attach an eSATA drive to our workhorse computers, generate all this ProRes and Proxy media, copy the FCPX project files, then plug that drive into our MBP's and go. Right?

Actually, you should NOT be putting large projects on your OS drive at all. Huge no-no. Create your Events and Projects on a secondary drive. With eSATA you'll see really great performance. When we do transcodes in FCP X, ProRes 422 or ProRes Proxy, we're getting WAY FASTER than real time on a 2009 MBP with a FW800 drive. a 3 minute clip will transocde in less than a minute.

I did a job last week on a brand new MBP with a T'bolt drive, and it was faster than my MacPro with it's internal drives.