Feature doc finishing challenges

Sorry this is long, but I don't know where the problem lies: in my workflow, FCPX, or my editing machine, so this is all of it.
I am finishing the edit on a feature doc (± 80 min). I have 1.7 Tb of video clips, spread over 5 events. The structure is basically chronological: following 4 people over the course of a year, ( www.onegoodyear.com ).
My editing machine is sort of bottom of the barrel: MacPro 1,1 (2x2.66GHz Dual-Core Intel Xeon w/12gb RAM. The graphics card was upgraded to one recommended at the beginning of FCPX, but it has only 512 mb of vRAM.

Main problem is progressive slowdowns. After a couple hours working, it will just come to a halt, and I can see it executing commands I gave it 3 minutes ago.

The director visited each of these growers periodically, and filmed the garden activity that was happening that day, and they talked while the camera rolled. I have been working with Compound Clips as edited scenes of each of these visits, dropping them into a longer 'Act', and breaking them apart (I 'm not assembling sequences with CCs) Of course I reedit them in the context of the whole, but this method leaves the CC intact as basically the selected edited version of that day's activities.

So each Project is an 'Act', 12 to 20 min long.

I have been duplicating and renaming Projects for versioning, creating a new Project folder and dragging the new versions in (this dragging in the Project Library is so slow and tedious).
I use Event Manager (brilliant, simple app!) to just work with the newest folder.

So there's the basic workflow; here are the questions:
How to create the assembly?
Create new Project
Open 'Act01', Select All, Copy
go to New Project, Paste.
OK, except that FCPX has a problem with single frames (don't know where I read it, but try dragging a Gap down to zero: doesn't happen, one frame is left. (Workaround: I usually use T to roll the adjoining clip over it, and then shorten it up again, but you can Lift the adjoining clip, extend it over the Gap, then lower it again into the Primary Storyline.)
I am finding that Copy/Paste to create an Assembly is a huge hassle! Almost every connected clip will be one frame too short. Extending 80 minutes of connected clips one frame each is a tremendous waste of time and frustration. Especially because I have already done a very precise edit. The ones you notice are the ones where the connected clip is over a Gap, and should butt up to the next clip in the Timeline.
I fixed a million of these gaps, and yet watching the DVD (albeit, not the finished version) I see them flashing by all over the place.
I don't trust that the connected clips are not the only ones that are not being created precisely from the frames I designated.

Each of these Projects/Acts has a 2-pop at the head and a sound upcut in the mixed track I get from my sound editor. So exporting and bringing back in a QT.mov is not the answer, as I see it, because I have to overlap the sound mixes and 86 the Beeps, and sometimes make a transition.

So what do I have to do to get this movie constructed and exported?
Is it software? Or is it equipment?
I was hoping to get a New MacPro, but now that won't be till end of Oct. if that.
I could buy a MBP, but I don't need another laptop, and I don't want to edit on one.(and don't want to use that MacPro $$)
Get more RAM and upgrade graphics card in this machine. Will that help?
TIA<
Gretta

Greta,

I edited a 3 hour long project for a client and had the same slow downs & hang-ups as you are describing 2 years ago (IIRC, 10.0.3 or whenever multicam was put back). That was when FCPX was not as mature as it is now so I feel the pain you are experiencing. I was editing on an early-2011 MacBook Pro 2.2 Ghz Quad Core i7 16GB RAM, 1GB VRAM, 7200 RPM HDD (before I upgraded to 512GB SSD). My system drive was a 750 GB hybrid SSD and a second 750 GB 7200 RPM hard drive replaced my optical drive. I was cutting the original media (non-transcoded h.264 DSLR footage).


1. COMPOUND CLIPS: The main problem I see in your workflow is the compound clips. When working with many of them in a project file, it tends to slow FCPX down. Even though you are breaking them apart the CC remains in the event, in essence, clogging up the pipeline. My solution is to create their own projects which I think you've already done which is causing more unwanted problems, like the gaps & single frame issues. I have not experience this issue. Solution to this is to export from thg CC timeline, more on that in #5.

2. EVENTS: Having multiple events should not cause any slow downs but as a precautionary measure, I like to work with one so I typically merge events. This may not be feasible for your project but food for thought. Also, I delete unused render files from events & projects often.

3. PROJECTS: Having numerous projects also slowed my MBP down before purchasing Event Manager X so hiding projects help. Project file sizes no longer affects speed, in my latest experience.

4. EQUIPMENT: 512MB VRAM is not enough for FCPX so that definitely adds to the slow down problem. Like many, you do not want to upgrade with the imminent MacPro release around the corner but you also have a deadline to meet. What are the specs of your current laptop and is it a MBP? I'm guessing that it's not or else you wouldn't say you could get one. If you go that route, I would peruse Craigslist for a 15" or 17" MBP quad Core i7 with at least 8GB RAM (16GB RAM upgrade is $140). Attach the current MacPro monitor to it. When the project is complete resell back on Craigslist. Market price is $1,200 to $1,500.

5. ASSEMBLY: Finalize each Act then export, reimport it and assemble your main timeline from these Acts. You can export the Compound Clips you've already created from their own timeline - keep in mind that it must be near the finalized cut to prevent having to export/import for each change.


QUESTIONS:
1. What is your project resolution? 720p, 1080p...?
2. What are your hard drive specs?
3. Are the events, projects and media on a separate drive from the system drive?
4. What is the original media codec? Are they transcoded? Are you working with proxy?
5. In FCPX playback preferences, is it better performance or high quality?

I haven't experienced slow downs lately since moving on to a Late-2012 iMac 3.4 GHz Quad Core i7 32 GB RAM, 2 GB VRAM with a Pegasus 12TB RAID 5. I've invested a lot of money to remove al the bottle-necks in my hardware & workflow.

You have many bootle-necks in your hardware for such a demanding project. The new MacPro will alleviate your stress but that's half of the solution. The other half is having a sufficient data storage solution like a RAID 5 for demanding projects (if you don't have one already). The key is freeing up all the bootle-necks from ingest to export.

I hope this helps.

BTW: There's a green rush here in Colorado & I'd like to do a piece on the same topic in the near future.

Not sure you have one but if you don't replace your system drive to an SSD drive.

Because of the larger amount off data FCP tries to keep track of the memory is not enough and the System caches memory to the drive. Using a traditional drive just slows things down dramatically for the system to write and read the memory cache.

Workflow suggestions when working with long timelines: don't display the frames and waveforms in the timeline when you don't need too. Use the lozenges view mostly. Zoom in and switch the display to work on details and than switch back. You can map timeline views to the keyboard for fast switching. Everything FCP needs to display will be eating memory. Switch to list view for your events. Again minimising the amount of video frames to be displayed. That said over a day FCP will still fill up the memory and the SSD will come into play.

Your problem with the 'missing frame' when pasting and copying I don't see here. The only time there was a problem was when I copied and paste clips with different frame rates to the source clip and destination timeline. Eg the clip is 25 and is copied from a 24 FPS timeline to a 25 FPS. I can't remember the details but I suggest you check the frame rates of your clips in relation to the different timelines you're copying and pasting to.

My machine Mac Pro 3,1 (2008) 16 GB, ATI 5770 1 GB Vram, Raid drives via eSata.

I'm right now editing a 1 hour doc. I originate from 50 events which are online all the time. (about 7 TB, ~100 hours of video and audio, 12488 items) I edited by scene each in its separate project (about 20 with average 3 versions = 60 projects online all the time) with multiple versions and just copied pasted all into my Master sequences which started out as a 90 min cut. Not one problem. Now 4 Master versions down (always cutting in the full length timeline) to 65 min. Yes a bit slower than editing a 4 min seq but totally fine to work with. I was afraid it wouldn't be but FCP is more than ready for prime time.

Thank you both for such a wealth of information!
I did not realize the CCs were creating a bottle neck by being in the Events. I did know not to use them unbroken in the project. I don't need any of them anymore, so will trash them all.
Discovered that render files have grown huge: will trash those as well.
I can see that I need to break the Projects/Acts at straight cuts, rather than the transitional places they are now broken. that way an export/import of QT.mov will not be so problematic. (like film reels)
Equipment, equipment, equipment…I like your 'temporary' MBP idea, altho' Craigslist is not the place here, as hundreds of laptops have been stolen recently from the tech college, and I have at least one innocent friend who proudly showed me her 'new' laptop, with inventory markings very prominent!
Anyway, a trip to the Apple Store in town.
Frame rate:YES!!! Some of this footage was shot 23.98, and I have been creating the new projects at 29.97, so that is most probably where the one frame issue comes in with Copy/Paste from one project to another. I was not very consistent or knowledgeable at the beginning of this project with Project Properties.
Re, Lozenges: I never (in 12 years) used thumbnails in FCP. Now I am addicted. I realized that the timelines look and work a lot like editing film on a Steenbeck: i could see thru the celluloid a ways to see where the next cut was etc. I liked the little visual color changes flashing by.
I appreciated FCP v.1 for its interface that really catered to film editors but I tho't vX had gone completely into the digital/metadata world.
but yes, I can work w/ lozenges, and will to ease the strain. (event browser always in list view)
and thanks, both of you for the info that you are doing long-form successfully w/FCPX. It is hard to find that community. I have purchased many tutorials and done so many freebies, but close to NO info on workflow creating verite. I don't have a script to work from, so I needed those edited CCs hanging out until I got the structure and order nailed down.
The biggest slow down has come in the last week when the director was here and we totally restructured the thing, creating many CCs to hold scenes we were temporarily removing from a project to move to another.
I will immediately implement all possible suggestions, and thank you so much, once again.

Yep it's early days for long form with FCPx. But FCPx was made for it!

A few workflow things I do:

I always import all footage into the Event meaning no reference. I don't see the point to have 2 sets of organisation on my system (Finder and FCPx). This also allows me to easily archive everything aafter I'm done. I basically treat Events like a footage library. Therefor I separate things like music, sfx, graphics into their own events. Global stuff I would need for for more than one film gets a separate event again. All this is geared so I can later access events for stock footage.

I let FCPx create proxies and edit with those. On the system I have it just seems to work snappier. Playback performance is improved and I get more realtime effects, which lets me avoid rendering altogether. FCPx keeps perfect track of proxies to original. I can at any time switch to the original if I want to check something in full res. And the proxies are absolutely fine to edit with both in quality judging and editorial judging. What is neat is that you can apply effects and they behave the same in the proxy as they do in the original (no scaling problems!) Brilliant really

I have a second computer monitor switched to display the viewer. This lets me avoid an additional video out card like Blackmagic or Aja. I have one if I need to. But using one slows down the system.

For exports of long master projects (QT export) I render the timeline. I haven't made up my mind yet if rendering actually benefits working with long timelines. I have a feeling it does. But I always delete all renders of previous versions of the cut.

And yes using thumbnails and waveforms in the timeline is a dream come true. Specially audio editing. When I'm editing the shorter scenes I always have them on. Just noticed in the long master they slow things down. FCPx has a brilliant way to map views to the keyboard. Therefor switching between modes is super fast and easy. And the keyboard has ore possibilities than the timeline menu. e.g. increase the waveform height and reduce the video height within the chosen overall clip height - and vice versa.

Happy editing

Just back from trip to local Apple specialists (not The Store), with invoice for 8gb more RAM, new graphics card, and an SSD.
Trashed render files, and most CCs. I experimented with trashing one (of several) that said they were being used in an Event or Project. That left me with offline clips in the Project. Not a huge deal for me to replace, but I do not understand how it is referencing the compound clip, rather than the original media. And why only some of the clips in that scene? Was it perhaps only referencing the CC for clips that I never adjusted after breaking it apart? Odd behavior, IMHO.
Fingers crossed for this coming upgrade. It will/should at least allow me to finish this film with relative ease.
I will report back…

Compound clips (or synched cips) behave as if they were original media.
Kind of like referenced QT files in the old days. Deleting one never deletes the original media, but if you used a compound in an edit and delete it in the event/project they'll disappear everywhere. But this doesn't delete the media they were made off. The other way round if you delete the media a compound is made off the compound doesn't disappear but just goes 'media missing'.

And yes please let us know how you are going with the hardware changes.