Multiple Hard Drive Workflow Recommendation?

Hello.

I'm working on a documentary and I have so much footage it spans a few 2 TB drives. I'm starting to build my rough cut in FCPX 10.2.3 and I was wondering if anyone had advice for file management. I'm editing portions of the story timeline as "chunks" and each "chunk" is it's own Library. Would it make the most sense to copy all footage to 1 big drive so I can access it all at 1 time or plug in all the drives each time I work or put the Library on it's own drive and copy footage to it as I edit, etc.?

I hope that made sense. It's much simpler when all the footage fits on one drive. I just want to avoid being redundant and be as efficient as possible.

Thanks!

djzimmie wrote: Hello.

I'm working on a documentary and I have so much footage it spans a few 2 TB drives. I'm starting to build my rough cut in FCPX 10.2.3 and I was wondering if anyone had advice for file management. I'm editing portions of the story timeline as "chunks" and each "chunk" is it's own Library. Would it make the most sense to copy all footage to 1 big drive so I can access it all at 1 time....

There's nothing wrong with having footage spread across multiple drives. FCPX even allows multiple drives to comprise a single library. However each of those are more complex and require additional care in backup.

Since in documentary the final story is often discovered during the editorial process, you ideally want all the content concurrently available for searching and skimming, which would mean in one library. There is no hard upper limit to library size but there's a soft limit where after 5,000-7,000 clips it may get slower -- depending on the drive and computer performance.

Using FCPX 10.3.4, I'm currently working on a doc which has about 136 hr of H264 4k in 5,000 clips comprising about 5.5TB for media and 2TB for proxies (about 7.5TB total). This is all in a single library and the media is on an OWC Thunderbay 4 32TB four-drive RAID-0 array. The media is backed up on various drives and the library is backed up via both Time Machine and Carbon Copy.

The media for this is all within a single folder tree, and the proxies are in their own separate folder tree, not within the library. It mostly works from a performance standpoint but the Event Browser can be a little laggy at times building thumbnails in filmstrip mode. This is on a 2015 top-spec iMac 27 but I'm moving it to a top-spec 2017 iMac 27 which has about 2x the transcoding performance. I tested a 12-core Mac Pro D700 but it was actually slower for our workflow.

If putting chunks of your material in separate libraries works for you, that's OK. That does potentially solve some scalability issues but IMO it makes searching and organizing more complex.

Re editing portions of the story timeline as "chunks", this could imply a stringout or assembly edit approach. That has long been a traditional method but FCPX is designed for a different approach, which is organizing, rating and keywording the content in the Event Browser before you even touch the timeline. You can do it either way, and admittedly the keyword/rating approach may take some mental adjustment. However much experience has shown that using FCPX the way it was designed can produce faster results.

Here is an older MacBreak Studio which shows some of the concepts:



Ripple Training has an updated complete training class on this:

www.rippletraining.com/products/final-cu...-in-final-cut-pro-x/

Re physical media management (as opposed to logical management using metadata) they have a separate class on this, updated for 10.3.x:

www.rippletraining.com/products/final-cu...-final-cut-pro-10-3/

I don't have any relationship with those guys, I'm just mentioning this because I found it beneficial.

Hello.

This was by far the most helpful post I've received yet. Since I'm such a newbie, I have a couple follow up questions and I apologize in advance.

Full disclosure, I had no idea what I was doing when I started filming so finishing this documentary is kind of a Frankenstein retrofit. I've got footage from various cameras of various qualities and a lot of it was shot either out of order or before the script was fully wrapped up. (Before you yell at me, I've been yelling at myself for about 18 months, no need to, ha.)

It sounds like I need to buy an 8Tb drive, copy all the footage to it, start over logging / organizing it and then assemble the timeline based on my edit. Do you have any tips for that? I'm going to watch the MacBreak video now. Sadly I'm on 10.2.3 and the Ripple training is for 10.3. (Long story but I need to stay on Yosemite for audio software compatibility issues).

I'm guessing I want to pick up 2 RAID drives, copy all the footage, work from there, back up to the second? I've used Carbon Copy cloner before. Have you used Chronosync? I've seen some posts about it working well as it can incrementally back up while you're working on the project.

Thank you again. This has all been a painful but rewarding learning process.

djzimmie wrote: ...I had no idea what I was doing when I started filming so finishing this documentary is kind of a Frankenstein retrofit. I've got footage from various cameras of various qualities and a lot of it was shot either out of order or before the script was fully wrapped up...

Footage from multiple cameras is no problem in FCPX. Being shot out of order is typical for many productions, esp. docs. Often with docs there is no "script" per se, only a treatment or an outline. Documentary is one of the greatest strengths of FCPX -- it allows taking all that disparate content, rapidly skimming, rating and keywording it.

djzimmie wrote: ....It sounds like I need to buy an 8Tb drive, copy all the footage to it, start over logging / organizing it and then assemble the timeline based on my edit. Do you have any tips for that?...

Buying an 8TB drive might not be necessary, provided your current drives have adequate capacity and you can back it all up. With FCPX the physical layer of organization is totally separate from the logical or metadata layer of organization. You could later "consolidate" the material to a single physical location using FCPX tools, say for archiving.

That said, IMO it's easier to keep track of and back up a project's physical media if it's all in a single folder tree.

djzimmie wrote: ..." start over logging / organizing it and then assemble the timeline based on my edit. Do you have any tips for that?..."

Depending on your current progress into the endeavor this might be useful. Done the FCPX way, it feels at first you aren't making progress -- you are endlessly skimming through material, marking favorites, rejects and keywords, maybe refining these as you go. Note these are *range*-based, not clip based. This is a profound difference between FCPX and other software. Apple page on selecting/marking ranges: support.apple.com/kb/PH12532?locale=en_US&viewlocale=en_US

As you plow through this phase, a producer might even ask "show me what you've go so far", and it appears you have nothing. That's because you aren't building a huge assembly as the 1st step. Then when you finally get the content organized, the timeline phase goes *much* faster. In past documentaries I have spent months in the organizing phase, then was able to 95% complete the timeline within two weeks, and at a high quality level. That is how efficient the FCPX approach is.

With FCPX it's best to work in the Event Browser 1st and extensively -- do not be rushed getting material on a timeline. When the content is finally organized properly, THEN start building the timeline.

The traditional assembly method can be very inefficient and time consuming. Superficially there's a timeline there but every time you need a well-matched cutaway or b-roll shot, it yanks you out of your creative reverie to search. This happens over and over -- creative death by a thousand cuts. With the FCPX method, it's already organized and findable within seconds.

The FCPX way also can prevent creative block, which especially happens in documentaries as you sift through the material looking for the story. With the content well organized in the Event Browser, rejected clips de-clutter extraneous stuff (yet are still available), favorites highlight good material, keywords link main shots to b-roll, etc. Oftentimes just browsing this organized version lets the underlying story jump out at you.

djzimmie wrote: "...I'm guessing I want to pick up 2 RAID drives, copy all the footage, work from there, back up to the second? I've used Carbon Copy cloner before. Have you used Chronosync? I've seen some posts about it working well as it can incrementally back up while you're working on the project..."

That is one way. I use both Time Machine and Carbon Copy for backups. Time Machine does incremental AND continuous back ups of everything you do, with ability to roll back to many different check points of single files or the entire system. Carbon copy does incremental backups, so each backup only copies the files changed since the previous backup. However that forms a single snapshot of the disk state, with no ability to roll back to previous backups.

Any backup disk can go bad, and neither Time Machine nor Carbon Copy can do a validation of the backup. You can run Disk Utility First Aid on the backup to ensure the file system itself is OK but this doesn't scan the entire contents. I used Acronis backup on Windows for years and this is available for Mac. It can do validation scans of the backed up content. However I'm happy with TM and CCC.

CCC can do a Check Sum to verify the data is good, but that takes a lot longer to do, and copying data between drives is pretty stable these days. Depends on your personal preference. But a Check Sum for data integrity is about all any backup software will do.

Personally I'd never trust Disk Utility to inform me is a drive was going bad. I've seen it repair drives over and over, each time claiming it was good, then the drive dies. It isn't monitoring sectors and blocks. But it is a good idea to run it weekly on all your drives. Plus run "Disk Warrior" on all your drives monthly.

You'll increase overall system efficiency a little bit using one larger RAID over many smaller drives, too.

FCPX.guru wrote: CCC can do a Check Sum to verify the data is good, but that takes a lot longer to do, and copying data between drives is pretty stable these days...Check Sum for data integrity is about all any backup software will do..

Current versions of CCC checksum the data by default, mainly intended to check for infrequent memory-induced corruption during backup. Disabling this requires using the following terminal command:

/Applications/Carbon Copy Cloner.app/Contents/MacOS/ccc -g ThisMacHasECCRAM bool YES

Software like Acronis does a lot more than this. It re-reads each data block in the backup file and validates the internal data structures. There is no built-in procedure for either TM or CCC to do this. You can do a Disk Util First Aid on the TM or CCC backup, and that will check file system only, not the data itself.

You can use a 3rd party disk scan utility to read the entire backup drive and verify no read errors. Most people don't do that.

Yet this can be vitally important because when you commit to a full restore, whatever is on your current disk will be wiped out. If the restore gets to 80% then errors out, you lost both your current disk and the backup is no good.

One solution is making yet another "safety" backup before you try the restore, but most people don't do that and they may not even own sufficient disk drives. Since neither TM nor CCC backups have a built-in feature to validate the backup integrity, when you commit to a full restore (maybe for a relatively minor problem) you are rolling the dice.

With today's hard drives, usually nothing goes wrong but you never know. Imagine this case: during a large Finder drag/drop, the user's hand trembles and the data goes to parts unknown. The user isn't aware of the Finder undo command and they spend time poking around trying to figure out where the files went. In frustration they call Apple who tells them "You've got Time Machine, just roll the whole thing back". The TM restore requires wiping out the system drive before the TM restore. The backup HDD was apparently good at backup time but subsequently developed some bad blocks. The TM restore errors out at 50%. The user now has no data and no good backup. Had they validated the backup with a disk scan (or had TM had this feature built in) the problem would have been avoided.

As you say, disks are generally very reliable and I personally don't do a disk scan on the backup except in rare conditions. I don't use Acronis on Mac, just TM and CCC. However I always have multiple backups of multiple types, so if one fails I can try another. In a case like the above I'd take a safety backup before trying the restore, but I have lots of hard drives.

I run Disk Warrior on all drives including backups monthly, it checks everything.