MASSIVE Library. Any suggestions?

I'm about to begin editing a MASSIVE production (100+TB) with hundreds of hours of multi-cam 4K interviews. I'm using the new 16 inch M1Pro Max and a for the hard drive an external G Drive Shuttle 72TB. (Project is not yet 100TB but I imagine it will reach there eventually). Wondering If anyone has ever experienced issues due to a library becoming too big? Is a huge library Is more likely to become corrupt? Could It be more safe to break the project apart into multiple libraries and then edit between them? Open to hearing any thoughts from those with similar experience. Thank you.

I would say the most important thing would be to separate the Library from the media, meaning have the media external to the Library.
From my experience it is best to have the Library on the internal SSD drive, setup the cache location in a separate folder on the internal SSD as well for quicker deletion in case there is a hiccup with corrupted cache files. And store the media external to the library on external drive(s).
I have currently a Library with 30TB of external media (stored on several external drives) and the corresponding Library of 1.7GB on the internal SSD. Works without a problem.

I've worked on a 9TB library (not mine) with all of the media in the library itself. Tons upon tons of events inside said library. Many projects in each event. It's sooooo sluggish. Working is fine but it gets bad when it backs up, which it seems to do a lot and it takes forever. Launching said library or even quitting Final Cut is also an exercise in patience. I have suggested having the media separate from the library next time for this director and not so many events.

In my experience, with the new hardware we have at our disposal now, location of media, library, and library size have little effect on speed or file corruption. Your workflow should decide media location and library size.

The hardware specs of your computer, codec, and throughput of your storage determine speed. Fast CPU, GPU, enough RAM, optimized media, and a fast internal or external drive should provide smooth editing and grading. A M1 Pro/Max should have no problems with a large library, whether it's on an internal or external drive.

I keep multiple large libraries up to 50TB with managed media (original media stored in the library) on a QNAP RAID capable of 1000+ MB/s read and write. Using an M1 Max, I have no slow down or corruption issues because of library size.

A large library and media location - external or managed media - become more of an issue if the workflow requires libraries to be moved or shared with more than one user. As a single user, I don't need to collaborate and share libraries, so I go with the convenience of managed media and self-contained libraries and not having to worry about losing links to the original media. When I back up libraries, the media in the libraries are also backed up automatically. Peace of mind.

For my next project, there will be multiple editors, so it makes sense to keep the libraries small and store the media outside the library. The smaller libraries can be easily copied and shared, while multiple editors can access the external media on the QNAP.

Why do you think it's the number of events that is slowing down the project? Rather than the amount of data and computer/hard drive speed?

No way I'd put all my video eggs into one library basket where one I/O hiccup would stall or maybe sink the whole ship. Maybe one library for each mega multicam interview and other similar content segments.

Similar to building a house. No need tripping over shingles, paint cans, or carpeting when you're still working on the foundation. Keeping the incidentals out of the way until such time as they're needed makes it easier for me to focus and not have to scroll through endless projects and events.

KavanaFIlms wrote:

I'm about to begin editing a MASSIVE production (100+TB) with hundreds of hours of multi-cam 4K interviews. I'm using the new 16 inch M1Pro Max and a for the hard drive an external G Drive Shuttle 72TB....issues due to a library becoming too big? Is a huge library Is more likely to become corrupt? Could It be more safe to break the project apart into multiple libraries and then edit between them?...

- Ideally all material for a given production should be in a single library -- if the material is interrelated and if you'll need to query all the material. FCP has no ability for cross-library queries. 
- If you ever need to pass projects or clips between libraries, this should be done within an event. Sam Mestman discusses this from 06:30 to 11:00 in the below video, but you may want to watch Sam's whole talk from 02:00 to 11:00. He demonstrates this on a Lumaforge NAS but it's not unique to a NAS.

- During the production phase, rigorously mandate each camera and recorder has correct time of day, verified each production day.
- Decide up front what the distribution frame rate will be and make sure all cameras and recorders are set to that frame rate. This is to avoid rate conforming artifacts or the CPU cost of optical flow rate conforming.
- Use the same color and gamma profile on all cameras. Do not try to intermix (say) S-Cinetone and S-Log3. 
- Make sure all shooters know the exposure targets for their camera and gamma profile. E.g, on Sony S-Log3, the target is 61% IRE on a 90% reflective white target.
- Monitor all audio.
- Where possible use 32-bit floating point recorders such as Sound Devices MixPre-6 Mk II. This greatly reduces problems from clipped audio or low levels. However clipping can still occur on a wirelss lav Tx/Rx link if not set properly:  www.bhphotovideo.com/c/product/1503005-R..._6_ii_6_channel.html
- White balance all cameras on each scene and make sure all videographers know the procedure for each camera.
- Make sure all videographers use the 180-deg. shutter rule (e.g, 1/50th sec shutter for 23.976, 24 or 25 fps material).
- Make sure all videographers clean all lenses and sensors for each shoot. Removing sensor/lens debris in post is expensive. A Rocket blower is a good tool:  www.bhphotovideo.com/c/product/259157-RE...cket_Air_Blower.html
- To the extent possible, use jam-synced timecode. Even mirrorless cameras can be manually jam-synced using free run and preset timecode. E.g, after jam syncing all other cameras and devices, set mirrorless cameras to "free run" TC and "preset". Set the preset to 1 min in the future. Observe the free run TC on the big cameras, and at top of minute, press "preset" menu on mirrorless cameras. That will usually be within about 1/4 sec. All NLEs inc'l FCP can sync on timecode. You can also bulk update timecode on all clips from a  given camera using the 3rd-party utility QtChange:  www.videotoolshed.com/handcrafted-timecode-tools/qtchange/
- Be aware than 23.976 fps is by definition NDF, so if other cameras are shooting 29.97 or 59.94 (by default DF timecode), they should be set to NDF if timecode sync is a priority.
- Use external "leave files in place" media, do not copy it within the library, however...
- Do not import tree-oriented media with "leave files in place", e.g, AVCHD, Sony MP4, etc. Re-wrap those externally using EditReady first:  hedge.video/editready/benefits
- Upon offload, rename files or add a 5-digit "uniqueifier" suffix so that all files are globally unique within the entire production. This gives a simple reference during post and can avoid some FCP issues with spurious duplicates.
- Files can be batch-renamed using Finder: 
- There are other sophisticated renaming tools such as "A Better Finder Rename":  www.publicspace.net/ABetterFinderRename/
- Periodically duplicate the lean library in Finder. If media and cache are outside the library it will be small and you can keep multiple backups. This provides another level of backup beyond the FCP auto backups stored in /Movies/Final Cut Backups.
- Periodically export an XML of the entire library and of the main project. That provides yet another backup type.
- Have a separate dedicated SSD for the library and cache
- Do not use extremely large numbers of events or projects. Each event and each project is a separate SQL database with performance overhead. 
- There is no hard limit to FCP library size. I've edited a documentary on FCP that had 8,500 clips, 220 camera hours and 150 multi-cam 4k interviews. That was on a 10-core iMac Pro and several 32 TB OWC Thunderbolt arrays. The main issue is # of projects (aka timelines) and # of events. I suppose if you were over 10,000 clips that might encroach on some limit. If you have less than 30 projects and less than 20 events, on modern hardware that is probably OK.
- I have the M1 Max MBP 16 and it's really fast but even it can struggle on some "difficult" codecs. If collaborative work is a priority consider using transcoding from 4k to 50% ProRes Proxy. M1 Max has dual ProRes decoders and encoders. The fastest path is use ProRes 4k acquisition, then transcode on M1 Max to 50% ProRes Proxy -- it is incredibly fast on that. Currently there's a performance bug on FCP unique to Apple Silicon whereby transcoding 4k to 50% H264 is quite slow.
- Redundantly back up all media on site. Do not transport media in the same bag or same vehicle.
- More comments on keyword organization here:  www.fcp.co/forum/4-final-cut-pro-x-fcpx/...multicam-clips#90333

joema wrote:

- Do not import tree-oriented media with "leave files in place", e.g, AVCHD, Sony MP4, etc. Re-wrap those externally using EditReady first.

Why do "tree oriented" video files even exist any more? It's 2022. What's the advantage of such a structure? Seems like a huge pain.

Also the tips above say "Ideally all material for a given production should be in a single library" and it also says "Leave the files in place". So, which is it?

Redifer wrote:

...Why do "tree oriented" video files even exist any more? It's 2022. What's the advantage of such a structure? Seems like a huge pain.

I don't know the detailed history, but here are some comments: Historically the file formats used by video cameras has very rudimentary, often using FAT-32. They couldn't store data using NTFS, HFS+ or APFS which have transactional protection.  The container formats themselves often don't support rich metadata within the file header, which led to using sidecar files.

Whether using a single file or tree-oriented sidecar files, camera manufacturers have tended to have a closed, secretive and proprietary viewpoint about metadata in their file formats. E.g, even though various 3rd-party metadata tools can extract some metadata from Sony MP4 and MXF files, only Sony's Catalyst Browse utility can get complete details. The manufacturers tend to squirrel away non-documented but important metadata in fields called "maker's notes". Then the 3rd-party metadata tools like ExifTool, ffprobe, etc must reverse engineer those.

It may be that using sidecar files give less compatibility issues as the proprietary fields of camera manufacturers are changed. E.g, say a new camera adds an internal ND filter. If they try to add acquisition metadata showing ND setting to a flat file, they could solicit a standards body to add that field in the next update to the container format (which might take years), or just jam it into some "maker's notes" field, but that conceivably could break some downstream workflows if software didn't expect some new metadata field to suddenly appear.  It could be more expedient (but messier) to just add it to a sidecar file which are not central to video I/O and less likely to cause problems.

I think there was a movement in the more pro-oriented MXF container standard to disallow proprietary manufacture-specific extensions. MXF as used by our Sony FX6 cameras also has sidecar files but the MXF container format itself is much richer than MP4.

Redifer wrote:

...Also the tips above say "Ideally all material for a given production should be in a single library" and it also says "Leave the files in place". So, which is it?

By "in a single library", I meant don't split frequently-referenced material from a single production across multiple libraries. I didn't mean *within* (as in copy to library). E.g, if someone asked "which library is the Jones interview in?", you wouldn't say "it's not in any library, it was imported using leave files in place".