Possible massive data loss due to new consolidate command. Help required

Hi all.

I've just been consolidating media to a NAS using the new consolidate command (changing from managed to external). Some of the way through my network connection was interrupted, and the clips have vanished. They are no longer in me original media folder inside the library, nor are they in the new location, or they are but they are not complete video files and so can't be read. It's about 80 percent of the project media which won't relink.

I had it backed up on Time Machine, but apparently time machine isn't quite smart enough to deal with massive libraries and despite my best efforts, I can't restore the full 200 GB library, only 35 GB.

Does FCPx just do a dumb move with consolidate? No copy and check before deleting out of the original library. If so, its a pretty poor design. Anyone have any suggestion for here I might find my media?

Don't shut down your computer.
Contact Applecare and/or send feedback via www.apple.com/feedback/finalcutpro.html

- Ronny

Thanks so much Ronny. Its not a good day. It's good to have a something to grab on to.

While I'm here: can you tell me why you are not able to retrieve the files from your backup drive? In cases such as power failure or an interrupted connection while moving files your backups are your best friend.

- Ronny

My time machine backup reports the library is 200GB, which is the size of the full library, however when I drag it over to copy I only get 35GB coming back, I've managed to find some footage in the __async copy folder, but not all of it. I'm having a look for hidden files through terminal which is all I can think of.

That's why I suggested not to turn your computer off. Since OSX 10.5.1 the OS caches files before moving them. Bt as this was a huge move I'm technically not savvy enough to be able to guarantee you that you will be able to recover everything. Backups are extremely important in case things like these happen.

When you go into the Time Machine drive can you find the Event folder(s) with all your media in there? Instead of dragging the entire .backupdb folder you should go into the folder and just copy over the Events.

- Ronny

Not sure when and if you want to try some data recovery, but I've used Data Rescue 3 on a 2TB that I accidentally formatted and got almost everything back.

www.prosofteng.com/products/data_rescue.php

It takes a while and use a much larger drive than what your recovering. This program recovered things that had long been erased and written over. You might be in better shape since nothing was written over yet. It's worth maybe a call to these guys to see what they say before you get it. Good luck, I know the feeling.

Ronny's advice is spot on.

The other important thing not to do is use the computer until you hear from apple. If you don't write to the drives the more likely you will get most or all of it back. I'm assuming this was your only copy?

When a file is deleted off a drive only the file header is removed and the file is still there but to the drive is blank space waiting to be written over. Software like Data Rescue has saved my hind more times than I'd like to admit. The downside is you may not get the file names back.

Good luck, and don't lose hope.

We were writing the same thing at the same time! Spooky!

Darren is right, but if the name is lost, it'll be a sequential list: M0062.avi, M0099.avi (or whatever format it is DV, MOV etc...)

Thanks guys, I'm running a data rescue scan scan now, to see what comes back. The DP had backed up the epic mags, so I'm lucky there. I'm taking care not to write to the drive til I hear something back.

Nothing is more fun than sorting through a bazillion files after an unintentional format. Been there, not fun.

Hopefully since you didn't format the drive your file names may very well come back intact.

Slightly off topic, I can't say enough good things about the data rescue people. I would be happy to tell the story of them saving a major day of shooting on a doc series I was working on. But first, I'll wait with crossed fingers that Seamusvfx gets his stuff back.

Ha, I didn't mind the sorting after a while compared to the chilling feeling of possible losing tens of thousands of pictures. Data Rescue and Carbon Copy Cloner are your friends.

I've also followed Ben Balser's backup strategy ever since. He has it posted somewhere, but it's very efficient.

Well, I just relinked the last of 400 clips. It's taken me all day. Thanks to everyone for there suggestions. For future reference here are my thoughts on the consolidate tool:


1: Don't use the consolidate tool to move a managed media to external media on different drives. You're better off copying the media in finder and relinking, otherwise, should you lose connection to the destination drive for any reason, you lose your footage.. Not just the one file being copied either. As fcpx copies multiple files simultaneously you will corrupt most of your media.


2: Time machine alone isn't necessarily going to save you. In Time Machine it appeared the library was completely backed up with all original media in place. When I went to restore the backup however, it had also changed the media to aliases leaving nothing to fall back on. Get a better backup plan.

3: If you should disconnect, stop what you are doing, don't close anything, go into the library through show package contents. There is a folder called __asynccopy or similar. this appears to contain the files waiting to be moved, but it is purged automatically at some point. I managed to get one card back from this folder.

Apple seriously need to address this I think. A tool that does a simple file move is asking for trouble in professional media management. If you were doing it manually your would copy, verify and then delete. The software needs to do the same.

Here is Ben's link that he gave me.

fcproxuniversity.com/FCPro_X_University/...loner_For_FCP_X.html

SeamusVFX Wrote: "Apple seriously need to address this I think. A tool that does a simple file move is asking for trouble in professional media management. If you were doing it manually your would copy, verify and then delete. The software needs to do the same."

I'm really glad you got your footage back. Completely losing hard work and moments that are next to impossible to recreate is something I wouldn't wish on anyone.


The consolidate feature is not intended to be a replacement for Carbon Copy Cloner or Shotput. It's the same thing as 'command-dragging' files to another drive to move them. I agree with you that manually moving everything is much safer, but it's not as convenient.

But...

There is no software that will ever be perfect enough to protect you from needing backups. Every editing platform assumes you have backups. I used to have an HVX200, by dragging and dropping the P2 files from the cards I would often get clips with one duplicate frame. I blamed the camera until I found out you should clone the p2 cards to disk images. If I ever saw a dupe frame again, I reimported the clip into FCP (5!) from my dmg and presto! it would be perfect again. Having cloned backups of all camera files is mandatory. Having multiple ones is strongly advised.

That article Don posted is great and shows how easy it is to be safe. I'll get off my high horse named Backup Giddyup now. I'm glad it worked out for you and I hope this is the last scare you have. I've been there.

I entirely agree with you Darren. There are a number I things I could have done to make recovery easier, and my data more secure. I'll definitely be reading that article and putting it into practise.

I maintain that it's really poor design to have a mechanism that can so easily cause so much damage. I'm a big fan of fcpx, but I think that's a severe flaw. Finder copies between volumes by default for good reason. Other media mangement run checksums on the files they copy and then delete. Apple recommends using fcpx for media management in their documentation and the he tools need to be robust and they need to be forgiving.

In this case i removed the network cable too early because I was in a hurry to get away for Christmas, so that was my fault, but presumably if fcpx crashes (as it sometimes does ) while something is transferring in the background the presumably the consequences are the same. There is no way in which that can be said to be acceptable.

Once I did a 'move files' (copy and delete) with the finder that I forgot was running. Closed the laptop, and yanked the drive. There was this huge file that was gone from both drives. (A master shot done with only one take) I went into obscenity autopilot after I realized that it was gone.

It is a good idea to add something like 'safe consolidate' as an option.

Just to follow up on this, Apple got back to me for more information and have said they are actively investigating the issue. Let's see if something turns up.

So this has been fixed as part of the 10.1.1 update, which is pretty impressive of Apple. I just ran a test on it and here's how it works now. When you consolidate to an external drive all the media in your library in moved into a folder called __AsyncCopying. Is then copied to your external media and automatically deleted from your library on completion.

During the copying process symlinks are added to the external folder, looking at all the original media in __AsyncCopying. This means that your media will come online when you reconnect your external drive and all you need to do in rerun the consolidate command and the operation will complete.

The only thing to be aware of is that appears that in the event of a connection interruption media from __AsyncCopying may not be automatically deleted, so to free up the hard drive space you may have to do that manually.