FCPX slow, every basic function is extremely slow :(

joema wrote:

I need to review everything done so far and get a fresh look at it. I will post any info here. Dave if you have time, feel free to help or comment on any of the above. As soon as I re-verify the best representative library and project, I'll post a link to that.

Yeah, I haven't seen the XML, so I'll defer to you, Joe, on this. I just wanted to comment about how complex trying to troubleshoot FCP can be. So many known variables, as well as unknown variables...

Slaughter, I am reviewing the problem history. Please correct or add as needed:

Before the problem you were on:
Imac Pro 2019OSX Monterey 12.6.1FCPX 10.6.5Libraries on external SSDs.

- Are you still on Monterey 12.6.1? If not state what version.

You updated a library for a project largely completed under FCPX 10.4x. You then updated the main project from 720p to 1080p. There were no immediate errors but the updated event and main project under 10.6.5 runs very slow.

Had the library/events/project completed under FCP 10.4x ever been updated before this? Was the library version still 10.4.x?

If the library version was still 10.4.x, when was the last time it was used and on what hardware? I'm trying to determine how to test this. Do you have any backups of the library from when it was 10.4.x?

Can you look at the FCP auto backups (by default in /Movies/Final Cut Backups), see if you can find a library backup corresponding to just before you upgraded. In Finder do right-click>Show package contents. Select CurrentVersion.plist and press the space bar to view the file. What catalog version is listed? That is the FCP database version of that library backup. Does that match what you expect?

Please make a separate file backup of the entire Final Cut Backups folder, in case we need that for latest study. 

If you find an FCP auto-backup of that library which is just before the upgrade, please zip that and send it to me.

There are several possibilities:

(1) The version upgraded modified the database schema making it inefficient for your unique case. E.g, if it added or removed SQL indexes, added columns, etc. it could cause a slowdown, maybe only under certain conditions.

(2) If the previous FCP version that touched that library was 10.4.x and the upgrade went to FCP 10.6.5, it's possible the FCP code itself is slower for some reason in this specific case.

This is an important problem because several other highly-experienced editors have encountered things vaguely similar on roughly similar timelines -- although not apparently tied to an update. Your case might be our best chance to understand this.

>Before the problem you were on:
>Imac Pro 2019OSX Monterey 12.6.1FCPX 10.6.5Libraries on external SSDs.  

Yes

>Are you still on Monterey 12.6.1? If not state what version.

12.6.6

>You updated a library for a project largely completed under FCPX 10.4x. You then updated the main project from 720p to >1080p. There were no immediate errors but the updated event and main project under 10.6.5 runs very slow.

Correct

>Had the library/events/project completed under FCP 10.4x ever been updated before this?

Yes.  I started working on this under whatever version of FCPX was back in …. circa 2018.

>Was the library version still10.4.x?

Not back in 2018…. if I am understanding the Q correctly.

>If the library version was still 10.4.x, when was the last time it was used and on what hardware?

Basically this project goes back to 2018 and whatever version of FCPX was around at that time.
There has been one change of imac (from a standard imac to the 2019 imac pro I presently use.  This changeover
took place around June, 2021. There has been one change of external HD type –from 3.5inch HD in a Promise thunderbolt bay, to SSD via USB3. 

>I'm trying to determine how to test this. Do you have any backups of the library from when it was 10.4.x?

TBC.  I’ll take a look.

>Can you look at the FCP auto backups (by default in /Movies/Final Cut Backups),
>see if you can find a library backup corresponding to just before you upgraded.
>I’m not 100% sure exactly when I upgraded to FCPX 10.6.5 but I’m sure I can find something which evidences this… TBC

>Please make a separate file backup of the entire Final Cut Backups folder, in case we need that for latest study.

>If you find an FCP auto-backup of that library which is just before the upgrade, please zip that and send it to me.

Note that I am still under 10.6.5, and have not upgraded to 10.6.6 yet.  Largely out of disillusionment that none of
the UI glitches I have been complaining about have been addressed by this update.
I’ll get onto your specific requests asap. Thanks again.

Slaughter wrote:

....

>If the library version was still 10.4.x, when was the last time it was used and on what hardware?

Basically this project goes back to 2018 and whatever version of FCPX was around at that time.
There has been one change of imac (from a standard imac to the 2019 imac pro I presently use. This changeover
took place around June, 2021. There has been one change of external HD type –from 3.5inch HD in a Promise thunderbolt bay, to SSD via USB3.

>I'm trying to determine how to test this. Do you have any backups of the library from when it was 10.4.x?

TBC. I’ll take a look...

Since this performance problem apparently did not exist prior to the library update, if at all possible I need a copy of the library prior to the update. This is the library only, no media needed. A recent performance problem was traced to certain internal library SQL characteristics, which enabled us to backtrack and deduce what user-visible FCP event & project construction caused it: fcp.co/forum/4-final-cut-pro-x-fcpx/3813...nts-opening-projects

Several other reported performance problems might be very loosely related in terms of certain event & project layout, but we have not closely examined the internal composition of those. It is very time-consuming and difficult. The above problem was only resolved because DaveM and I spent a huge amount of time on it.

Last night I used some command-line SQLite utilities to examine your library. These are MacOS x86 versions but they run fine on Apple Silicon. These allow examining the size of the SQL tables and indexes inside the library. The tools are free and available here:  www.sqlite.org/download.html

Background: Each FCP library is internally composed of various SQLite databases, each in a separate file named CurrentVersion.fcpevent. There is one for each project and one for each event. Each database contains eight tables, three of which are of interest. They are named ZCOLLECTION, ZCOLLECTIONMD, and Z_3CHILDCOLLECTIONS, and there are various indexes on each table. There is no documentation for the table layout or organization. However, it's possible to deduce a few things. ZCOLLECTION is the main table. For each clip, edit action and effect, it has several rows describing those. ZCOLLECTIONMD (MD=MetaData) is mainly a BLOB (Binary Large Object) table, with some columns to aid lookup and cross-referencing the other tables. The binary data is not documented. Z_3CHILDCOLLECTION seems to track parent/child relationships, such as between multicam and compound clips to the parent clips, and maybe from those to each instance range used in a project.

The output of the sqlite3_analyzer utility shows your "All Noises" library has some unique properties. I didn't notice this before because we normally focus on the project (aka timeline) properties since that's where problems often exist. In your case the event database is unusually large (about 670 megabytes) by itself. Inside that database, the single table ZCOLLECTIONMD takes up 82% of the space, which is unusual. 

Study of the BLOB column from that table indicates each row is a binary plist (bplist) datatype. I think Apple documents the bplist format, but not FCP's application-specific interpretation. That is where I'm currently at -- trying to decipher this for troubleshooting purposes.

However, as a very preliminary assessment, there is something unusual or abnormal about your event's layout. There is a very large number of clips, but that by itself does not explain what I'm seeing on a SQL level. Is it possible the clips themselves have been edited outside of a project via Clip>Open Clip? If this was done to a lot of clips, that might partially explain what I'm seeing.

Dave if you want me to send you any items you can email me at joema4 at gmail dot com. I know you are busy on "real work" (so am I).

> I need a copy of the library prior to the update.

I’ll see if I still have one on a backup drive…
IfI can find one, be warned that the project timeline there will be horribly organised.

>Is it possible the clips themselves have been edited outside of a project via Clip>Open Clip?

I’ve never done this and didn’t know such a function existed!

Further to an earlier query about complexity of timelines within other projects within this library.  Yes, “All Noises 1080p” is easily the most complex – noting there is a 2^nd iteration of it in this library (its 720p form).  In a different library I am editing a 100 min timeline which is substantially less complex.  The endless beachballs and odd requirement to cross over to youtube in Brave browser to toggle play/pause to ‘unstick’ things does not happen there.  None of the shorter, less complex timelines / projects are beleaguered in this way, unsurprising given their brevity etc.

I have been using Brave for 6 months with no ill effects.
I asked around before using it and was told that it avoids the problems caused by Chrome.

Ian Brown wrote:

I have been using Brave for 6 months with no ill effects.
I asked around before using it and was told that it avoids the problems caused by Chrome.

Yeah, I've had zero probs.  Like I said, it's not that Brave *causes* the long beachballs - they occur whether I have a browser running or not.  The oddity is that Brave + youtube + hitting play / pause *unsticks* the beachball and allows me to continue working.  With this one project, Brave + youtube are now an essential part of my workflow.  Darned weird.

I'm not surprised that this project is exclusively problematic.  As described in previous notes, it's easily my longest running project, the most complex project and it has been forced through OS upgrades + FCPX upgrades. As well as going from 720 to 1080.  Anyway, as mentioned above I need to get some previous back up data to @joema which I plan to do by this weekend.

It's always good to check things like the Chrome browser issue, FCP preferences, etc. However I have five Macs and Slaughter's library has problems on all of them.

The library is not structurally damaged. I used a SQL tool and ran PRAGMA integrity_check which produced no errors. The problem is likely in the application-level data used by FCP and how the FCP software code uses that.

I'm still examining the exact makeup of the library on an internal level and a user-visible level. The ZCOLLECTIOMD metadata table in the event (not the project) is consuming an abnormal amount of space. Normally the event's SQL database is quite small relative to the project databases. I think that is an important key.

If the event's SQL database is huge and bloated for some reason, that would slow down most operations.

Today I wrote a Perl program to decode the BLOB binary field in that table. As suspected, it is in binary plist format. Converted to XML it's a bit more readable, but similar to fcpxml, it's hard to understand.

Whether that library state existed before the upgrade or not is one key item we're trying to determine. But two things get changed during the upgrade: (1) The library and (2) The FCP version. It might have gone from 10.4.8 to 10.6.5; if we can find a pre-upgrade version of the library that will tell us. Maybe the library was in this odd state before the upgrade but 10.4.8 ran OK on that. Maybe it's a performance regression in FCP that only happens with this unique library config.

Or maybe the upgrade itself did something to the library. If we can get a pre-upgrade copy of the library there are SQL "diff" tools which will compare the two and show any schema changes. I can also examine the SQL tables for datatype changes and data page consumption.

@joema
Into your dropbox, I've just uploaded 2 previous libraries containing this project, along with an explanatory text doc....

Slaughter wrote:

@joema
Into your dropbox, I've just uploaded 2 previous libraries containing this project, along with an explanatory text doc....

Updating the 10.4.10 version of your library to 10.6.6 results in a fair number of rows added to the main three SQL tables. I assume that is normal, but I've never studied it before. 

The oldest version of FCP I have is 10.5.2 on both Intel and Apple Silicon. I tried multiple times to upgrade your 10.4.10 library to 10.5.2. Sometimes it would work, and sometimes it would hang perpetually during the upgrade. However even if the upgrade was successful, FCP was prone to hang if doing almost anything - clicking on the "All Video" smart collection or a drag/drop of several clips on the main timeline.

So I don't have a machine and MacOS version old enough to test whether the library actually ran without problems on 10.4.10. But 10.5.2 is not a huge upgrade and it doesn't work well there, on either Intel or Apple Silicon.

I re-ran a bunch of SQL integrity checks on each project and event in the library. Those all return no errors. The library is not structurally corrupt at the SQL level, the problems are with the logical FCP data content or how the FCP software uses that. We really should have a library-specific integrity checker, similar to Lightroom's catalog checker. 

I'll have more to report tomorrow.

Thanks for the update!

I did more testing of your library on a top-spec 2019 MacBook Pro 16 running FCP 10.5.3 on Ventura 13.4.1. That was Intel and the library upgrade from 10.4.10 to 10.5.3 shouldn't have been that great. It was still super sluggish and prone to hang.

I exported a project XML from 10.5.3 and imported that to Resolve Studio 18.5 beta 4 on my M1 Ultra, and it ran very well. Filter commands on the Media page were lightning fast and timeline drag/drop was fast. It would periodically show a spinning wait disk for 1-2 sec, but not often.

I collected some more information about debugging the underlying cause and will pursue that tomorrow.

Oh, this sounds promising... but don't worry, I won't hold my breath!  Really appreciate this investigative work.

Any updates @joema?
No worries if you're caught up in actual work, though!

Unfortunately, I will be busy on actual work for about two weeks. Sorry, nothing I can do about that. Summary:

Your data is not corrupted that I can see. I don't think it's a hardware problem or a plugin. Your problem is in two general areas:
(1) Extreme slow performance, possibly related to extreme number of compound clips in the Event.
(2) Extreme slow performance on timeline, possibly related to same thing but there might be other factors. 

A couple of other customers with vaguely similar situations simplified their timelines or reduced the number of compound clips and that (combined with hardware upgrades) seemed to make it usable.

Your problem is especially difficult because it's not just the timeline. You could try cutting the timeline in half or quarters, and work on the pieces. In my tests that helped some but I believe the Event Browser problem still existed.

You reported it as suddenly manifesting after the library upgraded from 10.4.10 to 10.6.5. I cannot test the 10.4.10 version of your library due to dependencies between XML versions, FCP and MacOS versions. I upgraded it to 10.5.3 (the oldest version I can easily run) and it still happened. I tested both Intel and Apple Silicon, both on Ventura 13.4.1.

I ran a SQL Diff utility before and after the upgrade, and the upgrade process does add a significant # of rows to the main tables, but I don't think that by itself is responsible. See attached output from command-line sqlite utility which shows the page counts (not row counts) before and after the upgrade. I did some upgrades on other libraries and they also added rows/pages, but I didn't have time to test a wide variety.

Thanks, no worries, let's see what might emerge in a few weeks should time permit.

Breaking the project up into pieces wouldn't really be a solution for me because it's a feature.  Psychologically and intuitively I need to be able to work on it as a feature, rather than as a series of pieces, otherwise it will feel like a very creatively compromised process.  I can't really work like that.

Reducing the number of compound clips is probably not practical in terms of use of time.  Here's why:  FCPX in the first instance forces users to create more than an ideal number of them.  Case in point:  When you add a fade transition to a clip with a ken burns effect applied, the fade transition changes the way that ken burns effect behaves.  So you have to compound the clip with the KB effect before applying the fade.  This is something I've had to do innumerable times.   The other reason reducing the no. of compound clips is not looking practical is that doing *any* work in this time line at all causes beachballs. Maybe 1 in 3 or 4 mouseclicks does it.  If I have, say, 120 compound clips to export as printed files and then reimport, thus eliminating a compound clip, that's about half a dozen+ mouse clicks or steps to doing that per compound clip.  Which will each produce, conservatively speaking, at least 3 beach ball instances of anywhere from 5 seconds to 10 minutes each.  Which collectively would amount to a range of maybe a few, up to 18-20 hours of waiting for the beachball.  Just to remove those compound clips.  Pretty daunting!

There's the additional problem of random clips turning black, requiring a total deletion of all render files and then a reload of the whole timeline (and this happens about 50% of the time I start editing this timeline).

So - aside from any potential debugging solutions you might turn up in a few weeks, it seems my best bet is to painstakingly, piece by piece, manually copy and paste this time line into a brand new project in a brand new library (on the same SSD, of course) and hope to god that process recreates this feature but not the buggy properties that lie beneath it...

You can possibly export a project XML and import that to Resolve Studio. I did a few brief tests using your project; it seemed to work and the performance was much better (at least on my M1 Ultra machine). I don't have your media to test, but the problem seems independent of media.

Here is an older article from Ripple Training that summarizes the procedure and limitations:  www.rippletraining.com/blog/final-cut-pr...resolve-15-and-back/

I mention the above because you need some solution with the potential to work, however laborious. On FCP even if you copy/paste timeline fragments to a new library, there is no guarantee that would work. I tend to doubt it.

I know this is an important feature documentary, so I'm doing all I can. But due to my normal work I'll be very limited on this problem for several weeks. I will probably have to do a multi-week field assignment around the end of July, so I'll be gone then also. 

@joema
Thanks - no worries.

> You can possibly export a project XML and import that to Resolve Studio.

I take it you mean take it to Resolve and then export a new XML from that application and see if this provides a clean XML for FCPX to work with?

No, I don't think round-tripping the XML to Resolve would work. I meant a one-way migration via XML. From my limited testing, it seems to handle your timeline, but without data it's hard to tell. Resolve is probably faster for this case because it doesn't have to manage a magnetic timeline.

I once moved a complex 1-hr timeline from Premiere Pro to Resolve Studio, and that took about 1 week to get all the details fixed up. The free version of Resolve has some performance limits but the Studio version removes those.

Another issue might be your iMac Pro. I had a 10-core Vega 64 iMac Pro with 64GB for years, and it was pretty good (back in 2017-2018). It was sluggish on any Long GOP codec, and the single-core performance wasn't great by current standards.

Your 1.5 hr timeline contains about 3,500 clips, 1,000 cross-dissolves, 1,000 secondary storylines, 900 color correction effects and 215 masks. The containing event has 1,372 compound clips.

Your timeline has sluggish aspects on my M1 Ultra, but I don't have problems with the Event Browser locking up during queries. I can zoom into the timeline and trim clips. Yes, if I try to lasso and drag 20 clips on the timeline it will beachball for 5 minutes. But if I avoid the known problem areas, it's sluggish but usable. I think your current experience is much worse than that. This implies that hardware alone might solve part of the problem.

If you want to use the current timeline "as is", you might need a top-spec M2 Ultra Mac Studio. It is vastly faster than any iMac Pro, and significantly faster than my M1 Ultra Mac Studio. 

@joema 

Just imagine how happy I'd be if I broke the bank to spend $$$$ on a new mac only to find these issues are not hardware but FCPX bugs... and this project still beachballed to heck and back.  So yeah, I think I'll have to keep on being financially sensible for a little while yet, ha!

I'm also not keen on learning a whole new platform.  As little as I enjoy its glitches and poor updates, I'm an FCPX user, not a Resolve user....