Backing Up Projects When Using Hard Links

I'd like to know what others are doing for their backup solutions. Right now, I've got about 30TB of active and semi-active project media on my QNAP server. I've been using Carbon Copy Cloner to create regular 1:1 backups of the server to a simple 35TB USB drive array. I'm running into the problem of the backups being too large for the backup drive. I think the reason for this are FCPX Hard Links.

I keep media inside libraries, and many projects pull from multiple footage sources. For example, one of my clients has about 200 projects pulling from around 6TB of original media. On the QNAP this works fine, because Final Cut behind-the-scenes voodoo keeps the actual size-on-disk of all the projects to the size of the original media (6TB) via Hard Links – even though all the projects with their duplicated internal media might calculate to be almost double that.

This is a convenient way to work when fast, shared storage isn't especially cheap. The problem comes with backup, or more accurately, with backup programs recognizing the space savings of hard links. Is anyone using a backup solution that plays nice with this space saving feature of FCPX? Looking for a good working solution. Thanks.

Hi there, any luck with this? I'm having the EXACT same challenge with Carbon Copy Cloner as well...

Thanks in advance,

MA

This is not a Carbon Copy Cloner problem. My guess is that your backup drive is not formatted as HFS+, which is required for hard link support. If the destination filesystem doesn't support hard links, each hard linked file will be disassociated from the original file and will become a copy on the destination. As a result, your backup set will consume more space on the destination drive than on the source drive.

Actually this entire issue can be prevented by adopting a different workflow strategy. When you work on shared storage we usually recommend NOT to keep any media inside your Libraries, exactly to prevent things like this from happening. When you keep your media outside of the Library package, media will never be copied between Libraries and as a result you will have no media duplicate issues when creating backups of your media and your Libraries onto an external storage device.

- Ronny

Hi Ronny,

You are always so generous with your time and energy, you helped me with FCPX 10.0 many years ago with that trick where you could simply exchange the current.event file to collaborate remotely, those were the days! All my drives are Mac OS Extended Journaled which I believe is the same as HFS+ Journaled if I'm not mistaken:

en.wikipedia.org/wiki/HFS_Plus

We have all of our optimized media on external drives. We encoded the original media to ProRes 422, pulled that in with “leave files in place”, so FCPX sees our optimized media as original media. This keeps the libraries lighter but we do include everything a remote editor might need, so inside the libraries are proxies, audio and pictures. It's a 5 year project with 2000 hours of footage so even duplicate proxies when backing up drives does break the bank if the space saving of hard links is not respected.

If we were to store the proxies, audio files and pics outside the library on the same drive in folders, would that buy us anything? Editing between libraries would still create the magic space saving hard links right? Or would the library package contents files then create symlinks instead which backups would have no problems with? We have tried working with external folders for proxies etc, but have run into other challenges.

We keep the proxy, audio files and pics inside the library package contents because we have multiple remote drives, we clone to them, everything is inside each library and editors can take libraries on the road then send back XMLs of their work. If we need more detailed project FXs etc, they then make their proxy media etc live in external folders, they zip the light media-less libraries, then post them for us as transfer libraries.

We did explore keeping the proxies, audio files and pics in external folders on our master system drive, but that hasn't worked well as cloning the drives and media means a giant relink on the remote drives due to the fact FCPX works with managed media. Frankly FCPX is so finicky about managed media, drive IDs and struggles with the amount of footage that we have, that those kinds of remote drive relinks have been a disaster. Proxies invariably lose there links and there are no tools for linking just proxies, plus other strange behavior and it takes forever to relink.

As always appreciate all your time and energy, curious what your take is now. Thanks so much and hope all is well in your world.

MA

Hi Ronny,

I think I found a setting in CCC that dealt with the issue in advanced settings:

TROUBLESHOOTING/run deletion pass first

Seems to have worked! Hopefully this solution works moving forward.

Thanks

MA