Poor Event Browser thumbnail performance due to MTS files

When working with a large library (3.42 TB, 7,000 clips), I have found there's a significant performance issue with the Event Browser in filmstrip mode if using many AVCHD .MTS files. This is not apparent at smaller library sizes. It happens even if the MTS files are a small percentage of the total library -- only 350 out of 7,000 clips in this case. All files were imported with "leave files in place". It happens whether using a single thumbnail per clip or a longer filmstrip.

The solution was remove the MTS files from the library and re-wrap them with EditReady, then re-import them.

Examination with the UNIX Dtrace tools bitesize.d and iosnoop shows the FCPX process is doing a large number of small I/Os when scrolling down in the Event Browser to build thumbnails the first time. To a degree that seems by design. Some of the I/O is to the media files and some to Info.plist files.

However when .MTS files are present the I/O rate increases tremendously and disproportionately, which causes halting, sluggish scrolling in the Event Browser. In some cases the Event Browser will halt for 30-60 sec.

Conventional wisdom is I/O related to video editing is dominated by larger more sequential reads, thus the general recommendation from disk array vendors is use a stripe size 128KB or larger. In my case SoftRAID uses a fixed 16KB stripe size which is very small, yet despite that advantage in small I/Os the performance issue was huge. A more traditional stripe size would probably experience even worse performance in this scenario.

After the .MTS files were re-wrapped, Event Browser performance was dramatically and consistently better. In some cases the thumbnails still require rebuilding as you scroll down but this happens much faster and the Event Browser never halts for long periods.

I/O histogram from bitesize.d and Activity Monitor during one of these periods when MTS files were present: joema.smugmug.com/Computers/FCPX-Event-B...-Perf-Data/n-M7bG7L/

Just using Activity Monitor, it's obvious FCPX was doing huge number of small I/Os. 184 MB/sec divided by 15339 IOs/sec = approx. 12 KB per IO. The reads/sec are far higher than is physically possible so most of those are probably cached by either SoftRAID or the filesystem. But some % will "leak through" to physical I/O which caps the achievable I/O rate. This in turn causes a huge disk queue backlog of pending requests and ultimately FCPX must just stop and wait.

Unfortunately I didn't save the iosnoop data (which shows the actual files involved) from the problematic case but it appeared I/O to the MTS files was abnormally high and repetitive, relative to the other media files.

This is normally not a problem on smaller libraries and I'm not sure what the threshold is. Even though FCPX supposedly re-wraps AVCHD on import, something goes wrong which impacts performance. In general I would recommend all AVCHD content be re-wrapped externally using either ClipWrap, EditReady or some other utility. This also makes it easier to browse at the Finder level.

I have not tested using proxy or optimized media for this case since the data is so large.

System details: 2015 iMac 27, 32GB, 1TB SSD, i7-6700K, M395X, 16TB OWC Thunderbay 4 in RAID-5 (12TB formatted capacity), SoftRAID 5.5.5, FCPX 10.2.3, OS X 10.11.6.

There was an earlier related discussion before I finally figured out the problem. There may also be an issue whereby Spotlight Indexing is required on large media drives to provide optimal FCPX performance: www.fcp.co/forum/4-final-cut-pro-x-fcpx/...in-large-event#78667

Note the Dtrace utilities cannot be used starting with El Capitan unless System Integrity Protection is first disabled: apple.stackexchange.com/questions/208762...o-get-dtrace-working