Project Archival Backup Strategies

I'm still wrestling with figuring out the best way to back up my completed projects so that I can restore them at a later date for revisions/updates.

Here's what I'd like to do:

For every project I have the following:
- a number of event folders in the "Final Cut Events" folder in the root of the media RAID of my edit system. When importing, media is copied to the event folder rather than referenced and optimized media is created where appropriate
- a number of project folders for each show/version for the job in the "Final Cut Projects" folder of the same media RAID
- a folder containing what I call "FCP X Job Files"... things like music, voiceover files, scripts, AE and Motion projects, still images, etc... on the same media RAID
- a folder containing raw camera archives of all shot footage (as they were copied direct from the camera cards).

As a rule, I back up my archival files to mirrored pairs of bare 2TB/3TB SATA hard drives mounted in a USB3 drive dock. I keep one drive on site and the safety backup I keep in a secure offsite facility. Multiple job backups are stored on each pair of drives, from multiple clients. Other non-video jobs are also backed up on these drives. I know this is not super ideal as hard drives are mechanical devices and subject to failure/degradation over time and with lack of use, and I want to move to an LTO solution (or similar) as soon as someone makes one that is feasible.

So, now the project is in the can. What I'd like to do:
1: Copy the event folder(s), project folder(s) , FCP X Job Files folder and Raw Camera Archives folder to the backup HDDs using the Finder. These files would all be stored inside individual folders on the root of the backup HDD that are named according to the job.
2: Do a comparison of the checksums on the backed up files and the files on the media RAID to ensure the copies were clean.
3: Mirror the backed up files to the 2nd backup HDD.

Now, the issue here is file sizes. For larger projects, especially those shot with cameras that require media optimization, I'm backing up a lot of data that could potentially be recreated by FCP X should I ever have to restore the project. So, here are my questions:

1: I see that many tutorials recommend using the "Duplicate Project" command from within FCP X to copy the project files from one drive to another. What is the danger of manually copying the appropriate Project and Event folders to a subfolder on another drive using the Finder (not the Final Cut Events & Final Cut Projects folders)? Is it an issue of absolute file paths? It's messy to keep all of my project/event files backed up into single Final Cut Projects and Final Cut Events folders on the root of my backup drive. And besides, when I need to restore the project down the road, the files will need to be re-copied to a working media drive from the backup.

2: Can I safely delete the contents of the "Render Files" folder in the Project folder on my backup HDD?

3: Can I safely delete the contents of the "Transcoded Media" folder in the Event folder on my backup HDD?

4: For DSLR footage (Canon 5D/1D), can I delete the contents of the "Original Media" folder when I have the raw camera card archives backed up? For files like this, should my best practice be to NOT copy these files to the Event folder upon import (i.e. reference them as aliases) to avoid creating the redundant set of original h264 media files in the first place? I'm worried about ease/reliability of relinking here when the project is restored.

5: For MXF footage from a Canon C300, can I delete the contents of the "Original Media" folder and expect to be able to relink/regenerate the files later if the project needs to be restored? Unlike DSLR files, the name/structure of the Original Media files are created upon import and do not directly correlate to actual files contained in the raw camera archive backup.

Any thoughts on this would be appreciated. The whole backup/restore issue is a major pain point for us after having gotten it down to a science in FCP 7.

1: Copying in the Finder makes a duplicate that has the exact same database ID (UUID), and FCP X will only open one Event or Project with a unique database ID. When it sees the duplicate database ID, it will warn you it can not open duplicates. Working inside of FCP X creates a new, unique database ID for the new copy.

2: Yes, you can. I'd do it from inside FCP X in the File menu, so the database gets updated to reflect there are no render files to track. But manually deleting them has never caused any issues that I am aware of.

3: Only if you have imported the original clips into the Event folder, and relinking to those works. Relinking is a bit flaky at the moment. I'd not trust deleting those just yet.

4: There is no real "right" or "wrong" way, just what works for your needs. If you've optimized media, and don't plan on using them ever again, then there is no need to copy them to the Event folder. Creating an Archive gives you a duplicate of the original card structure to archive, if desired.

5: What exactly would you relink to?

Here's some articles related in one way or another. There is a lot of flexibility in FCP X, so what you do exactly will depend on what exactly you want. I only keep Optimized media, trash the original files. Some folks want to work only with originals to save drive space and never optimize. It all depends.
Consolidation
Copy/Move
CCC Backups

Will be interesting to see how others approach these issues. I'd personally like to see an Archive function that culls out renders, analysis, etc, as per user configurations, in events and projects, zips them up, and puts them on a specified archive drive.

Thanks for the input, Ben.

BenB wrote: 1: Copying in the Finder makes a duplicate that has the exact same database ID (UUID), and FCP X will only open one Event or Project with a unique database ID. When it sees the duplicate database ID, it will warn you it can not open duplicates. Working inside of FCP X creates a new, unique database ID for the new copy.

So as long as I'm careful about not ever letting FCPX see both of those folders at the same time (i.e. not letting them both be in "Final Cut Events" or "Final Cut Projects" folders at the same time) I should be fine? My worry is that there is some sort of absolute file path thing lurking that will bite me in the ass down the road when I restore an event/project that was copied off a drive named "15TB Video RAID" to a drive named "3TB Mobile Drive".

BenB wrote: 2: Yes, you can. I'd do it from inside FCP X in the File menu, so the database gets updated to reflect there are no render files to track. But manually deleting them has never caused any issues that I am aware of.

That makes sense. Wouldn't I have to quit FCPX quickly thereafter to prevent it from starting to re-render those files in the background when idle?

BenB wrote: 3: Only if you have imported the original clips into the Event folder, and relinking to those works. Relinking is a bit flaky at the moment. I'd not trust deleting those just yet.

So as long as the "Original Media" folder is populated with actual files I'm OK? That would be acceptable. The size difference between h.264 DSLR files and optimized ProRes files is one of the biggest space-savers when backing up.

BenB wrote: 4: There is no real "right" or "wrong" way, just what works for your needs. If you've optimized media, and don't plan on using them ever again, then there is no need to copy them to the Event folder. Creating an Archive gives you a duplicate of the original card structure to archive, if desired.

It's all about how easy it is to take the just camera archive and recreate the steps to restore the project to where you left it versus the cost of storage versus the amount of time it takes to back up the data. With a pair of 2TB bare drives going for $180 combined with the increased transfer speed of a USB3 backup interface, I'm tempted to just back up everything.

BenB wrote: 5: What exactly would you relink to?

I see your point. Theoretically I could re-import the clips from the camera archives and it should create identically named QuickTime movies that I could then relink to - but that's an extra step. As above, here I'm tempted to just back up both sets of data. I'm not 100% confident about the QuickTime movies that FCPX creates for the C300 media - but it's only a doubling of the storage required as opposed to 4 or 5 times of transcoded 5D footage. And the camera archives are useful if I ever need to check the EXIF data that the C300 records but that FCPX doesn't seem to maintain.

So as long as I'm careful about not ever letting FCPX see both of those folders at the same time (i.e. not letting them both be in "Final Cut Events" or "Final Cut Projects" folders at the same time) I should be fine? My worry is that there is some sort of absolute file path.

I was referring to them being on different drives, not the same drive. But it doesn't matter what drive they're on, UUID is UUID. That's all Core Data really cares about mostly.



Wouldn't I have to quit FCPX quickly thereafter to prevent it from starting to re-render those files in the background when idle?

I always leave background rendering turned off. Most of the time I get great playback with render lines.



I'm tempted to just back up everything.

See this article.



...EXIF data that the C300 records but that FCPX doesn't seem to maintain.

FCP X doesn't show the EXIF data? It has the metadata fields for it.

BenB wrote: I was referring to them being on different drives, not the same drive. But it doesn't matter what drive they're on, UUID is UUID. That's all Core Data really cares about mostly.

Right. So as long as two folders containing an duplicated database UUID don't ever get placed in any "live" folder while FCPX is running (i.e. are not online at the same time) it shouldn't really matter if they share the same UUID.

BenB wrote: I'm tempted to just back up everything.

See this article.

I meant that given the lower cost of physical media and increased data transfer rates, there is a diminished return on the time it takes to selectively remove data from a backup set in order to save space.

BenB wrote: ...EXIF data that the C300 records but that FCPX doesn't seem to maintain.

FCP X doesn't show the EXIF data? It has the metadata fields for it.

No, FCPX shows limited metadata. Things like device model name, device ID... There's a field for ISO speed in the EXIF View tab of the Info section of the Inspector, but it is left blank. In Canon XF viewer I can see ISO, shutter speed, focal length, aperture, lens info, etc. This stuff does not show up in FCPX.