How Big Is Too Big?

When it comes to FCP X Libraries, I think I have the answer.

I'm working on a multi-year project that revolves around a large library of footage that is repurposed heavily across many finished videos.

The current Library is weighs in at 5.6TB, 4.6TB of which is original media. A total of about 8300 clips. With more to come. The project/event files originated pre-10.1.x. This is now in one Library file with probably 20-ish events.

At some point after the migration of the events to the post-10.1 "Libraries" paradigm, the external media folder got "messy". Some media was just dumped into the root of the media folder, while other media was stored in the new folder heiracrhy (i.e. MEDIA FOLDER ROOT/Final Cut Original Media/YYYY-MM-DD/filename.mov). Files were duplicated in the same media folder... but some were not. The media folder was reporting itself as being 10+TB... on an 8TB RAID with 2.xTB free (doing that weird pseudo-symlink thing that FCPX does).

I decided that I wanted to clean this mess up before adding the most recently shot footage for logging, sorting and keywording. The most logical step to me was to consolidate the media to a new media folder on another volume - and then copy the cleaned folder back when done and re-link. This, it turns out, is impossible. The consolidate process grinds for a while and eventually the machine crashes with an out of memory error. This is on a newMacPro with 32GB. Maybe if I had sprung for the 64GB it would squeak by, but I doubt it.

So, I decided to create 5 "equally distributed" Libraries containing varying numbers of events (segmented roughly by the size of the media referenced by the chosen events). One by one I (successfully) consolidated the media of these new "sub-libraries" to the same external media folder on a new drive. Then, I copied this newly cleaned external media folder to the original RAID (after deleting the "messy" external media folder)

That completed, I'm now trying to relink the massive "master Library" to this newly copied media folder. My first attempt started at 5:30 pm and was still going at 9am the next day - with a spinning beach ball, FCPX reporting as "not responding" in the dock and the FCPX interface stuck on the "verifying files for compatibility" dialog. A call to Apple Pro apps support later and I could not tell if it was frozen or just slow. The rest of the machine was still running fine.

Bailed on that process (because, you know, work) and started again on a spare machine (iMac, 24GB RAM) yesterday at around 2. Over 24 hours later and it is still sitting there. I believe it is not locked up, as I can put my ear to the RAID and still hear the heads reading data. And the machine is still responsive otherwise (Finder is OK, I can launch other apps)

So there you have my experience. I'm just venting. I know that I could just leave the project split up into multiple Libraries segmented in some logical fashion... but the ability to do text searches on the data that has been lovingly entered into the notes fields of the clips is a huge time saver. It would be drag to have to perform the same search once for each of the individual Libraries every time you want to find all the footage related to the keyword. But I may have no choice at this point if the media relink does not eventually successfully complete.

Hi Scott,

A painful story indeed. But unless I have misunderstood something, I think this is what you should have done:

Open the Library and set the Library Media location to a new "PRODUCTION MEDIA" folder on the RAID.
Then Consolidate the Library. This will take a minute or so.
Done.

I have done this many times before even with large Libraries and on systems with much less RAM than you have. Doing this you create a copy of all your media that are spread all over your RAID into a new organized media folder on your RAID, without taking up any extra drive space, and the copying is done in no time. How is this possible?

1. You have a Library that points to media that are spread all across your RAID.

2. You create a new MEDIA folder, you point the Library Media location to that folder and you Consolidate.

3. FCP X will physically copy all the media that are spread across your RAID into the new MEDIA folder. But because the media are on the same drive the media are just duplicated via Hard Links. This does not take any time and although you have duplicated all your media it does not take up any extra drive space either. It's a bit of MacOS/FCP X magic and it's hard to believe until you have done it once and it makes your jaws drop. When everything is done you can safely delete the old folders and files in Finder.

As I said I have done this before, and if you wish I can post screenshots to show how to do it and to prove that it works. But it's so easy to do that I would advise you to just try it.

- Ronny

Ronny,

Thanks for imparting that knowledge. I knew that they were doing some file allocation voodoo there, but i had no idea that you could delete the initial source of one of those "hard links" and the second file would still point to the raw data. I'll have to test that myself - no offense, I'm sure it works like you say, but I have to see it with my own eyes!

That being said, I don't know if that would have helped me. Unless the fact that I was consolidating the "mega-Library" to an second drive (requiring file copies) was what was causing the memory error. But there's still a good chance that I might have run out of memory even if i were consolidating to a folder on the same drive like you suggest.

It's also possible that there is a problem with the Library itself - it could be corrupt. Last night at around 5:30 the file relink process seemed to move on to a new stage - but this morning it's still going and the interface seems to be stuck. But the drive is still showing activity.

Check out the weird dialog box it's stuck on:
<a href="/ imgur.com/d9Ez0tw "> i.imgur.com/d9Ez0tw.jpg " title="source: imgur.com" />

2.9 million files? I have no idea what that's about.

That figure is really high, but this does not mean the Library is corrupted. I would let it continue for awhile and if it does not show real progress I would consider quitting and trying out the method I suggested.

There is a huge difference between consolidating to a different drive and to the same drive.

When you consolidate to a different drive you are copying real physical files. This always works in three steps: read/store in RAM, create temp file, write final copy. These processes require heavy RAM usage and read/write drive activity. This also requires a lot of free space on your system drive to create the temp files.

When you consolidate to the same drive you are not creating real duplicate files, you only create hard links and this requires much less RAM and no read/write actions.

- Ronny

Yes, I could see how the physical act of copying the files to a second drive could be creating the massive RAM requirement.

However, I'm "pot committed" at this point and can't even try your method anymore. I don't have a valid, media-linked copy of the full Library at this point. The only place that the media is linked up anymore is in the "sub-Libraries". So I'm forced to re-link the files or work from the sub-Libraries.

Ah, I see. Let's pray it works.

Good luck!


- Ronny

Scott,

Your screenshot with the millions of files relinking got me worried, so i did a test.

I have relinked 3046 files across drives in less than 4 minutes. It is absolutely not normal that you see more files than there are to relink. Your relinking process is in an infinite loop, and after some testing I also know why.

Here's the solution:

Cancel the relinking.

Relink again, and when you have pressed "Locate All" and you have pointed FCP X to the correct media folder, make sure you see the "Click Choose to verify matches for "x" of "x" files found" message at the bottom the Finder window.

If you click Choose before you get this message your relinking process will start but it will go into an infinite loop and the file counter will go up infinitely. This is a bug, I have already reported it.

If you don't get this message at all: just select a lower folder until FCP X says it has found the files, and then click Choose.

In my test I first selected the top folder for my media files and I did not get a message that the files were found. Yet I clicked Choose anyway, and after an hour it was still verifying 80 000 files while there were only 3046 to relink.

Then I canceled, I selected a lower folder until I got the message that the files were found. I clicked Choose and FCP X verified and relinked the 3046 files in 4 minutes.

- Ronny

Ronny,

You were absolutely right. Thanks, man. It was a bug and I was caught in an infinite loop.

The library is now relinked and the media folder is clean. I appreciate the help. I owe you a beer (or three).

I will report the bug on my end as well.

S

My pleasure, Scott.

Glad you have been able to finish the relinking and you are up and running again.

Next time you want re-organize a Library that has external media spread across a drive, just Consolidate the Library to a new media folder on that same drive. This will be done in seconds (35 seconds exactly for 3000 clips), it won't require any relinking at all and it won't take any extra space on the drive tanks to the hard link voodoo. You really must see this before you believe it

- Ronny