'purge' = snake oil or magic cure?? (speeds up FCPX)

Hi professionals & insiders,

hobbyist here. Lateley, I dropped in a SDcard with about 20GBs AVCHD, ~180clips.
Import, optimize, proxy - my routine, to give my under-powered MacMini 2.26 a chance to handle 1080/50p.

usually, I do that overnight, knowing, this is a time-consuming process.

Today, I did it by day - noticing: 2% after 2h ... oooops!

Just for curiosity, I'm no engineer, I launched Activity Monitor.
ugly: half my RAM blue = inactiv.
more ugly: 0.2MB free, 600MB swap ....

Someone told me lately, to type PURGE in Terminal, which will 'free' inactive memory.
5sec later, brave Mini told me 4.5GBs free - aaaand: the activity display starts to count beyond snail speed.

---

Question:
#1) Why does FCPX 'occupy' so much RAM on import? In my humble understanding, this just shoveling data from one drive to another, plus some minor indexing/read out of meta data (180 clips - that sounds too me not thaaaat much)

#2) Why doesn't it 'free' that RAM, when starting with activity No2, transcoding?? It still doesn't make use of the 4GBs free.... but FCPX transcodes noticeable faster. Activity says before 5% CPU usage, now 160-180%?

… or is it just me? (OS 10.84, FCPX 0. .... :huh:

Purge = a wonder??

Been dealing with this same issue for a while. Happens usually on clips over 4GB and the whole import stops. A major bummer when I decide to import a large number of clips and walk away for a few hours.

I don't have any solution though. I type purge and the system seems to speed up - but I've heard it does "bad things" to your system. I have yet to actually verify that anything "bad" is happening however...

FYI, I'm on an iMac 27" i7 with 32GB RAM and working off a Pegasus R4... so I'm pretty sure it's not the hardware...

thanks for your feedback!

my first assumption was some hodgepodge on my Mac, hadn't have a 'from the ground' install for years, tons of install/deinstalls of testing shareware etc etc.-

fcpguru wrote: … but I've heard it does "bad things" to your system. I have yet to actually verify that anything "bad" is happening however.....

I'm VERY shy with any Unix-ing … lacking complete ANY knowledge, what I'm doing there - 'til know, reading a lot about the purger command (and understanding little) I haven't stumbled upon 'bad' reports yet.

Just believe, not knowledge: for sure, WHILE an app is running doing any fundamental changes with the OS HAS to be … 'critical'. Lucky me, I'm in position of doing no time-critical jobs: if the import fails, I go back to square1, losing a day.-

Tom Wolsky gave me a reply, that this 'RAM wasting' is for years a flaw/bug in the OS, and is probably in focus with forthcoming updates .... imagine a superMacPro with 256GBs Ram running out of memory :woohoo:
____________

anybody else trying 'purge'?

Actually the purge command flushes the disk cache, not physical RAM.

man purge
"purge -- force disk cache to be purged (flushed and emptied)
Purge can be used to approximate initial boot conditions with a cold disk buffer cache for performance analysis. It does not affect anonymous memory that has been allocated through malloc, vm_allocate, etc."

So basically, it is clearing a segment of Virtual Memory from being included in the pie chart you see when using Activity Monitor. It does not increase system performance, and in a short time that pie chart will return to what it initially looked like, once the Virtual Memory (disk cache) once again mirrors physical RAM.

Huge myth that it frees up physical RAM, it doesn't. It is only a footprint utility for analysis only.

A CNET quote:
"Do keep in mind that while this command will appear to free memory, it will not increase system speed or RAM capacity. Rather, as I've mentioned in the past about other memory cleaner utilities, once you clear the disk cache or otherwise stress the virtual memory system to free up physical RAM, it will slowly begin to equalize and will soon return to its prior RAM usage levels. Therefore, instead of using the purge command or other memory management tools to "free RAM," it may be best to reserve these for determining programs' memory footprints and thereby isolating problems such as memory leaks, or determining if programs themselves are to blame for high RAM usage."

OS X does not do any RAM wasting, it has not gone on for years, it is not an OS X thing, no clue where Tom gets that information. There are apps that inappropriately allocate RAM, but tweaking the code fixes that easily. This can happen in any OS. Early versions if FCP X did in fact have a "memory leak", or allocated RAM in appropriately, but that has been fixed a few versions ago.

BenB wrote: Actually the purge command flushes the disk cache, not RAM.…

thanks, Ben, for joining with your expertise.-

I'm refer on advice givin' here e.g.:
osxdaily.com/2012/04/24/free-up-inactive...-with-purge-command/

the illustrations on that site reflect my observations here:
inactive 'memory' gets 'freed'.

… perhaps, my amateurish notations mixes Ram, memory, cache, whatever

What really excites me, is the 'real world observation', that importing large chunks of material gets a noticeable speed-boost by performing a 'purge' while doing it.

.... that's why I choose 'snake oil' in my thread title: it does work. Here. Somehow.
And (most) probably no good advice in a professional surrounding!

… what irritates me:
I've never found this as a 'on-your-own-risk-workaround'-advice concerning 'How To Speed Your FCPX'.- 

so, my first assumption was: just me. Freak effect.
but fcpguru above shares my obsevation too.-

.... next weekend, I'll start a new project, new imports - I will purge again, and will observate better, what's goin' on here.-

Again, the purge command has no effect on CPU/GPU processing speed. It frees up virtual memory, not physical RAM. The numbers you're reading includes this virtual memory. Run purge, yes, the numbers reflect to show only physical RAM, and doesn't include the virtual memory. But very quickly will go back to it's original reading.

If you experience system performance increase, it is mostly psychological, as RAM has no effect on CPU/GPU processing speed at all.

But, if it makes you feel better to use purge, it doesn't hurt your system at all, use it all you want without worry. Just to not use any third party utilities that claim to do this. They can clear inactive memory, which holds data the OS is using, and can do harm.

Here is a great explanation of what purge does.
reviews.cnet.com/8301-13727_7-57372267-2...nalyze-memory-usage/

And a great article on the myths of clearing RAM.
reviews.cnet.com/8301-13727_7-57348959-2...ea-they-claim-to-be/

#1) Why does FCPX 'occupy' so much RAM on import? In my humble understanding, this just shoveling data from one drive to another, plus some minor indexing/read out of meta data (180 clips - that sounds too me not thaaaat much).

#2) Why doesn't it 'free' that RAM, when starting with activity No2, transcoding?? It still doesn't make use of the 4GBs free.... but FCPX transcodes noticeable faster. Activity says before 5% CPU usage, now 160-180%?

While it's importing your clips FCPX creates thumbnail previews and waveform previews for every clip. Creating (or recreating) thumbnails and waveforms is known to be RAM intensive.

Creating Optimized and Proxy media this will also take up a lot of RAM as any form of transcoding requires RAM cache to store temporal information while the transcoded clip is being written.

The problem is that FCPX tries to do this all at once. There is no step 1 importing and step 2 transcoding. Your first imported clips are already being transcoded while thumbnails and waveforms are being created for the next clips that are being imported. All this can put a lot of strain on your RAM.

Another problem is that FCPX seizes all the available RAM in order to get everything done as fast as possible. This means that there is no buffer, which will slow down your entire system until your Inactive RAM (RAM allocation that has been used by the apps) gets swapped to Free RAM (RAM allocation that is available for new processes). That's inherent to the OS, like Tom said. And it seems Apple is working on it.

Until all this gets sorted out I think it is best to first import your files when you have many to import, then do the Optimizing to ProRes and Proxies from within the Event Browser. I think you will notice this goes much faster.

BenB wrote: … it is mostly psychological,

I'm afraid, it is! :lol:
... it is until now a single-shot-observation.
Have to repeat it under 'controlled conditions'.

next time, I'll take screenshots, using a stopwatch, avoid the pink pills & beer (wot??)

THANKS again!

You never want "inactive" RAM to be purged, ever! That is not the same as "empty" RAM space. It is RAM that is holding OS data that is not being accessed at the moment. But if you did clear it, the OS would go looking for it eventually, and you're hosed.

Solution, more physical RAM or upgrade the GPU. I have 16GB RAM, I never have any issues at all.

Ben, I expressed myself badly. While you were writing your reply I have edited the post.

ronny courtens wrote: …  I think it is best to first import your files when you have many to import, then do the Optimizing to ProRes and Proxies from within the Event Browser. I think you will notice this goes much faster.

AHAAA! That is some very handy advice!

THANKS, Ronny, for sharing your insights.
esp. hobbyists like me with underpowered hardware probably should follow your advice, of 'splitting' the import routine manually.

I'll try that … post my results next week!

I've heard this advice again and again that purge is a waste of time and does not work, but something must be screwy with my system because I'll come back when importing (just importing, no transcoding) a card and find it hung with the "Could not import clip, etc." message. When I check my activity monitor, every time the inactive memory is over 25GB and my computer has started to utilize the swap file. With 32GB of RAM, I don't think this should be happening. I'd say it might be corrupted cards, but it happens on all 3 of my cards and on cards that I have moved to the desktop as well. The only consistency is that it happens on clips over 4GB. However (strangely) the clips even though they give an error still import fine. It just halts the import process.

Back to the OP, the purge command does seem to speed up the import. At least my computer stops paging the HD after I use it. I have to believe that something is broken here...