Memory Leak in 10.4 when importing files

Is anyone else expering memory leak problems when importing files into 10.4. 10.3 with High Sierra did not do this but 10.4 does. I removed all plugins to see if that was the problem. No dice. If anyone has a solution, that would be terrific. Everything else seems to work but this rather crucial as one can imagine. TIA.B

Exactly what are you calling a "memory leak"? How is it effecting your work? That can be a vague term without specific symptoms.

It happens when I start an import. Memory rapidly increases until I run out of application memory. 10.3 did not do this. Everything else works. Removed all plugins to see if that was the issue. Nada. Sharing, rendering and outputting clips, not a problem but importing is a nightmare. Running High Sierra on a Mac Pro with 64 gigs memory and 80+ gigs on the SSD. Seems to me this is bug to the max as 10.2 & 3 did not have this problem.

80 GB on the SSD isn't much, depending on where your data and library are. Is it possible either library or data is on the system SSD, and you are simply running out of space during import? If the page file cannot grow due to you exhausting the disk space, it may appear to be running out of application memory.

I ran 10.4 on a Mac Pro with HS and had no issues.
I'd ask the same questions as Joema.

How are you determining that this is a memory leak? How are you monitoring your RAM?

anickt wrote: How are you determining that this is a memory leak? How are you monitoring your RAM?

If the OP responds, he can give an exact answer, but I think he got the error "Your system has run out of application memory".

However this does not necessarily mean an application memory leak. A more common cause is insufficient disk space, which is needed by the virtual memory system. Without sufficient free disk space on the system drive, it can produce that out of memory message.

A typical answer is "it didn't do this before 10.x", but maybe it was nearly out of space before, and 10.4 plus some newly imported media pushed it over the edge. FCPX is always needing disk space somewhere for cache files, render files, transcoded files, optical flow analysis files, etc.

The space consumption is not always predictable so it's best to have at least 20% free at all times -- and THAT assumes no media storage on the system drive. If any libraries or media is on the system drive, that can rapidly exhaust whatever is left.

It could be as simple as having background rendering on, and a library on the system drive will keep consuming more space as rendering automatically happens. As is commonly mentioned here, it's usually wise to leave that option off.

More info: macpaw.com/how-to/reduce-memory-usage-on-mac

Hi all, I'm suffering from this also. Here is what is happening with me.

Open FCPX 10.4 - create new library (2Tb SSD - 1.2Tb free) create new event, import media and watch the Memory consumption go from 1.2Gb to 32Gb (i have only got 16Gb - so work that out?!) and the Mac becomes unusable.

Background Render is off, the memory usage goes through the roof and it hasn't event started the Transcode or creating the Proxy media - this is simply copying the media across.

The footage from todays shoot is a mere 19Gb, i've tried different disks and attempted to not copy footage into library, it does the same.

So, 2 Macs - both unto date with the 10.13.2 OS. Sorry i cannot grab a screen shot of this happening - the Mac is unusable but the Memory pressure just keeps going up and up.

As a final footnote, i can only get to about 42% of import before the mac has to be reset, the Force Quit option doesn't work.

Any help - really appreciated. Thanks

Ive deleted the preferences, no improvement

I'm in the middle of a big project so I can't update to 10.4 or High Sierra, but lots of other people have without encountering this. It may be something unique to your codec or procedure.

What codec (IOW what file type from what camera) are you importing? You mentioned you tried not copying the media to the library. Could you try that again, IOW import with "leave files in place", do that from the import window not dragging them to the event in Finder. Verify it is accepting the "leave files in place", IOW the option is not grayed out in the import dialog. Also try deferring creating proxies or optimized media until after the import phase is finished.

Also have you updated to the latest Apple Pro Video Formats? support.apple.com/kb/DL1947?locale=en_US

Question, how can you leave files in place when pulling clips from a card or do you copy the clips in question to the desired place on disk. Also, I think I located the memory leak issue as I have a disk involuntarily linked to FCPX 10.4 that causes memory to go nuts whenever I do an import. When the drive is unmounted, imports work, as this time, as they should without the memory nightmare described in previous posts. I use Activity Monitor to track memory usage everytime I start up FCPX.

Question 2, is there a way to disconnect the offending drive from FCPX? besides unmounting? Am learning a lot from all the terrific comments on this thread. TIA.

Best.

Hi Thanks for reply

Im importing 1080p 60FPS from a Canon 80D - so not really rocket science.

I have tried the various methods you suggested, but i made purely by accident an amazing discovery. The MBP i have imports just fine, with background render enabled. So i enabled background render on the Mac, and the files came in no issues. Memory usage for FCPX was 560Mb. when i deactivated the background render, 32Gb of memory was consumed by FCPX leaving the mac unusable. So, there is a bug somewhere. All updates and the latest Codecs have been applied, for now i will leave background render enabled, so i can get this work out the door.

Thanks for your reply.

Heave High Sierra alone, its caused me many issues.

Dan

I've tested this over and over, and I don't get anything like what the OP or Dan are saying. Thus my questions are:

As before, what is the codec, and duration of the clips being imported.
Are any being Optimized or made to Proxy at all?
Is this Library on the system drive?
What PERCENTAGE of the system drive is free space?

As for memory going to 32GB, much of that is using Virtual RAM, which are hidden files on the system drive. That's not an issue normally.

I'd have the RAM tested, perhaps, also. None of this is normal behavior for a healthy system.

Clips are 10 to 60 sec in length. For 4k & HS - Sony XAVC S. No optimization, library resides on the drive where files are imported, NOT the system drive, 64 gigs ram on a Mac Pro, 80gis on the ssd system drive. What I see, as stated before, is a drive involuntarily linked to FCPX when importing clips with the spinning wheel in action, which indicates, AFAIK, the drive is being accessed by FCPX, which causes the memory FCPX uses to ramp up exponentially. When I unmount the offending drive before turning on FCPX, importing is no longer a problem. Question, is there a way to decouple the drive from X without having to unmount it?

Do you by any chance have a card folder (private, DCIM, etc.) that contains footage located on the root directory of your SSD? If so, that is likely why FCPX is automatically seeing that.

remoran wrote: ... how can you leave files in place when pulling clips from a card or do you copy the clips in question to the desired place on disk....

You cannot leave files in place when pulling clips from a card. In general I recommend copying all media to a hard drive and importing from there, but this is my procedure and there are other valid methods. If you import from the card it's important to make an archive of this, otherwise you have no backup: support.apple.com/kb/PH12710?locale=en_US

There are two issues with Sony XAVC-S files:

(1) XAVC-S is tree-oriented and FCPX will not import from a disk copy of the tree using "leave files in place" -- unless the .mp4 files are copied out of the folder tree to a separate folder. Doing so is considered poor practice because it discards the XML and other metadata files, however I haven't seen a problem so far.

Another option is re-wrapping all XAVC-S content with EditReady before import. This combines all metadata and media files to a single set of files which FCPX can import using "leave files in place". Re-wrapping is very fast but it will temporarily consume more disk space since both re-wrapped and original camera files must be on disk. However after the import you can theoretically discard the camera files and keep the re-wrapped files as backup: www.divergentmedia.com/editready

(2) Files from the Sony A7-series cameras start with C0001 and this resets each time a new card is inserted. This cannot be overridden or modified in any way, which leads to many on-disk duplicate file names. Even if these are safely kept in unique folders (e.g, project name, shooting day, camera name), after import FCPX does not handle XML for this correctly. So if you ever export and import library, event or project XML files, it will create spurious duplicate clip names. I haven't tested 10.4 to see if that's fixed yet.

Regardless of this bug, in general it's good practice to use unique filenames. It makes searching and identifying files easier. Before import I append a unique numeric suffix to all camera files (whether Sony or not) using the tool "A Better Finder Rename": www.publicspace.net/ABetterFinderRename/

remoran wrote: ....For 4k & HS - Sony XAVC S. No optimization, library resides on the drive where files are imported, NOT the system drive....a drive involuntarily linked to FCPX when importing clips...causes the memory FCPX uses to ramp up exponentially. When I unmount the offending drive before turning on FCPX, importing is no longer a problem. Question, is there a way to decouple the drive from X without having to unmount it?

By "drive involuntarily linked to FCPX when importing clips", I can't tell if you mean the drive is visible in the import dialog or the drive is shown in the FCPX library inspector. However -- in neither case should it cause a memory leak, but maybe that is happening.

For the import case, there is generally no "link" -- the mounted volumes are shown because FCPX did a "volume enumerate" API the same as Finder. But FCPX is not using or referencing any media on those. Only if you click on a volume name will it access that material.

Within FCPX if you click on the library name then look in Inspector at the top right, you'll see a heading "Storage Used for Media and Motion Content". That tells you what volumes this library's media are on. If you see your "involuntarily linked" drive, that means you have previously imported media from that drive to this library, and that media is still needed by FCPX. The only way to see what specific files is using the free utility Andreas wrote:

www.fcp.co/forum/4-final-cut-pro-x-fcpx/...ss-hard-drives#92661

If you simply disconnect the "involuntarily linked" drive and then examine all your clips in FCPX you will probably find red missing media thumbnails. In this case you'll need evaluate if those clips are really needed, and if not delete them from FCPX. If they are needed there is unfortunately no built-in method to selectively move externally-referenced files within FCPX from one drive volume to another. For externally-referenced files, the consolidate command is a brute-force method which mandates copying all media to another location simply to gather up a few files from a different volume. In many real-world situations it is not practical: support.apple.com/kb/PH15838?locale=en_US

You could delete the files on the inadvertently-linked volume from FCPX, use Finder to move the files to your media volume, then within FCPX relink the files: support.apple.com/kb/PH12701?locale=en_US

Having media on that other drive should not cause a memory leak, and in testing I will sometimes have media from many drives in one library. But IMO it's often the best practice to have all media for a given library on a single logical volume (which itself may be multiple drives via RAID).

If the "involuntarily linked" drive is a slow 5400 rpm USB bus-powered drive or a network drive, maybe it is intermittently failing or causing a permissions problem which activates an FCPX code path causing a memory leak. In general you probably want to (1) Verify whether you have any media on that drive needed by your FCPX library, and if so (2) Safely move it to the main media volume, delete it from within FCPX and relink the files from the new proper location.

An advantage to using all externally-managed media (IOW "leave files in place") is it produces a small, "lean" library which you can then easily duplicate from Finder, thus making a backup. That way if you experiment with something and it goes wrong, you can easily restore your previous library.

I have the same problem since 10.4. Memory usage increases continuously up to 16GB (total RAM) and then my system crashes. It is very annoying and also happens when I try to edit videos, not only when I import files. I have late 2013 MBPr 15". I'm now in the middle of reinstalling FCPX...

Just found this thread while researching this problem tonight. Had the same issue on my friend's 5k iMac when importing photos (and it turns out, ANY type of file import causes the problem) for a simple slideshow.

FCPX 10.4
OS is High Sierra
16GB RAM

Created a new library and imported 20 very low-res photos (typical file size was less than 100k). Photos appeared in browser more slowly than normal and iMac bogged down immediately. Tried selecting them all with Command-A and hit E to append them to the blank timeline. Took at least 30 sec to do this.

Managed to open Activity Monitor, and noticed that the RAM usage for FCPX was 20+GB (including HDD-based virtual RAM, one supposes). Needless to say, this was quite a surprise. Switched back to FCPX and quit the application (got an alert that background processes were being ended). Started it up again in that same library and everything seemed to be fine. CHecked on AM for a bit, and RAM usage for FCPX was steady. Was able to re-time a few of the photos, do some color correction, etc. All seemed good.

Went to import another batch of photos into a new event and the problem returned. We watched the RAM graph in AM ramp up sharply into the red zone. Photos did import OK but the iMac was unusable again. Closed the application once more.

Decided to trash FCPX preferences on next startup. DId that, and FCPX started OK (took longer than usual, but that's typical after trashings prefs). Tested a few things by making some basic edits, adding a title, a transition, a blur effect. All was good. Nothing out of the ordinary with RAM usage. Tried another import and problem was back again. RAM spiked back up and iMac was beachballing.

The photos I was importing were stored on a thumb drive (I was copying them on import to the new library on my internal hard drive). I also tested importing multiple file types, one at a time into a fresh library, from the same internal HDD and I still had the problem. There were two external HDDs connected as well, and we unmounted those in case they were somehow interfering. Same problem.

Regarding background render, it was on by default on that iMac. I did not try turning it off to test that setting - even though the opposite seemed to help the OP.

Eventually, we were able to import all the files (between multiple FCPX quit/restart cycles) to work on the complete project, and export it without any issues: A 36 min video of 764 photos set to music. Don't judge us, we're doing a simple slideshow as a favour for a friend! RAM usage maxed out around 5GB - nothing unusual there.

So, OP and others, you're not alone in this. I'm filing a feedback form with Apple to let them know what I was experiencing this evening.

I'm replying to my own post rather than editing it. We solved the memory spike issue on my friend's iMac!

It was exactly what dgwvideo had suggested. At some point, The contents of a GoPro micro-SD card had been copied directly to an external hard drive that is always plugged in. That meant that there was a folder on the root of that drive called DCIM. In the import window, FCPX was seeing that drive as a camera instead of a hard drive, and was scanning through 3TB of files to bring up all those thumbnails. Neither of us had noticed that drive name in the upper left of the Import window - and the spinning circle indicating there was search activity happening.

To make matters even worse, the same thing had happened on another connected external hard drive, but this time there was a folder named AVCHD in the root location, the result of a direct drag and drop from a camcorder. That drive had 4 TB of files on it. Double whammy!

So we solved the whole debacle by creating new root-level folders and dropping those camera-based folders inside them. Now those two external drives are no longer being detected as cameras - just plain old hard drives. RAM is stable, and FCPX runs like a dream.

Glad I found this post, and thanks to all of you for your great suggestions along the thread.

I don't know why but I've just read your reply and BINGO! Man, I owe you a beer!

This is what exactly happened to me. The funny thing is that I have 2x2TB external drives that were scanned in the background by FCPX. I had no idea why and how to get rid of them from the camera section on import screen.

Thank you so much!

I am using a Mac Pro (Late 2013) 3.5ghxz 6 - core Into Xeon E5 w 64gb of Ram and a
2012 Mac Book Pro 2.3 GHZ intel core i7 with 16gb ram
Both are running the latest version of FCPX (10.4.2) and the latest OS version (High Sierra 10.13.4)
Both are having the same issue that has only happened since updating to 10.4.2.
The issue is that when I try to import a large Sony file from an FS7 (40 minutes at 4K: about 106 GB), I run out of Application Memory or the Mac just hangs and stops importing.
In every other respect FCP is working perfectly.

AND HERE IS WHAT I FOUND OUT

I repeated the process of importing a large file into FCP while watching the memory tab of the Activity Monitor.
On Both Macs the memory used by FCP gradually rose to a higher and higher level and then settled back down again. On my Mac Book Pro when it reached 15 gb The MBP crashed. (I have 16gb of Ram)
On my mac Pro It got up to about 33 GB but then settled back down again to about 1.4 gb as it continued to import the file successfully.
While all this was going a senior Apple care adviser was watching my screen and has sent on the information to the engineers. He mentioned that there were several other people who had encountered similar problems.

So if its any comfort to people suffering the same problem, it is being attended to. Until it is fixed, and if you HAVE to import files, here is what I have found that works.
1. Turn off all other apps .
2 Import the files one at a time and when they start to transcode or create thumbnails, PAUSE this process .(you can transcode etc AFTER the files are imported. Transcoding does NOT cause the memory spike.
3, More RAM. In fact it should not be necessary, but in this case it is a temporary fix. (And more Ram is never a bad thing!!)