Import Speed

10.0.9 - I'm noticing some speed problems when importing multiple cards at the same time.

Example, if I import 5 32GB cards from a Canon XF100, one at a time, they each take around 13 minutes. That's around 65 minutes total.

However, if in the import dialog I tell it to import all 5 cards at the same time it's taking several hours (5 or 6).

Anyone else noticed this problem?

It also happens with AVCHD....

Yes, you're asking the CPU to process more data at one time. This can take it longer than if it just processes less data in smaller chunks.

Thanks for the reply Ben.

I don't understand how I'm straining the system by queueing (say) 10 files to import rather than just watching it import the first one and as soon as that's done, import the second one etc, because even if I queue them all FCPX still processes them sequentially and not in parallel.

The CPU use never goes above a few percent and HDD read/write never goes above 35MB/s on SSD!!!

It's not making sense and I don't see where the bottleneck is.

All background processing and analysis is turned OFF during import in both cases.

The primary bottleneck is in switching. If the system resources are going to be provided equatably then the system spends a considerable effort switching between devices. You see this in any file copying function. Simultaneous copying is significantly slower than sequential copying.

But my point is that even if I set them all up to copy at the same time, FCPX actually copies them sequentially, not in parallel (watch the little progress dial on each file and also that it's copying file x or x).

So why would it take soooooo much longer? A little longer I could live with, but 5 or 6 times longer is not reasonable. We're talking an hour or 6 hours on the same system.

What FCP says it's doing and what's actually happening are two different things. SD cards are outrageously slow, especially large cards. Every time you switch to a card the process basically stops down.

Take your cards, select them all, and drag them to your computer to copy them, watch what happens.

I'm not reading directly from cards (and especially not from SD cards). These are card images on the SSD drive.

So, we're going from one SSD to another SSD. The files themselves copy in just a few minutes. FCPX takes around one hour to import footage if I feed it one card image at a time. If in the import dialog I give it 5 cards to work on it's taking 5-6 hours. Since it's importing the files sequentially, something else is wrong.

I've trashed preferences and repeated this problem across multiple machines.

Repeat the same test on your hard drive. Trying copying all the content from all the archives simultaneously. Watch what happens. Sequentially is always faster. That's my experience.

Have no idea show trashing prefs will improve drive performance.

One of the other machines is running HDDs and not SSDs and the situation is the same.

Copying all the files (drag them in finder from one place to the other) takes just a few minutes on SSD and a little longer on the HDDs. It's not a file copy problem.

Also, FCPX "is importing them sequentially" and not in parallel. I'm not sure why you think it's happening in parallel all the time. It's not.

FCP's doing a lot more than copying. It's building new QuickTime wrappers for each file.

I know. And it's doing them sequentially.

So, why if I queue 10 things 10 times does it do it in an hour, but if I queue 100 things once does it take 6 hours? It's going through those 100 items sequentially in the same way it's doing the 10x10 sequentially, but it's taking 6 times longer.

Because it's trying to do it all at once and gets bogged down in switching. The same thing that happens on your computer. But I'm done going around this.

Tom Wolsky wrote: Because it's trying to do it all at once and gets bogged down in switching. The same thing that happens on your computer. But I'm done going around this.

When you look at the files being copied in the original media folder (which ones are links, which ones are real, which "one" is being copied) it's clearly doing them one at a time, not in parallel, but it's taking 6 times longer to do it. It's that simple.

Then don't do it.

When you import from a card you are in fact copying media from your card onto your drive. Copying involves reading and writing, which requires memory to cache the files. When the copying starts your system looks at the amount of data it will have to process and starts allocating memory for the process. The more data it sees the more memory will be allocated.

If the amount of data to be processed widely exceeds the amount of RAM you have, even when not all available RAM is in use, virtual memory allocation will occur: pages of memory will be written to your system drive. Writing to and reading from a disk is much slower than writing to and reading from your RAM. If the amount of allocated memory is so large in relation to the available RAM you have this can lead to disk thrashing.

Disk thrashing occurs when the system spends a disproportionate amount of time handling page faults and reading and writing pages, rather than executing code for a program,

Paging of any kind, and disk thrashing in particular, affects performance negatively because it forces the system to spend a lot of time reading and writing to disk. Reading a page in from the backing store takes a significant amount of time and is much slower than reading directly from RAM. If the system has to write a page to disk before it can read another page from disk, the performance impact is even worse.

That's why importing from multiple 32 GB cards a the same time will take much longer than importing them one by one. Unless, of course, you have 150 GB of RAM on your system

- Ronny

Thanks for the reply. Hmmmm...

ronny courtens wrote: When you import from a card you are in fact copying media from your card onto your drive. Copying involves reading and writing, which requires memory to cache the files. When the copying starts your system looks at the amount of data it will have to process and starts allocating memory for the process. The more data it sees the more memory will be allocated.

Except these files could have been copied to the SSD some time before, with most of the ram remaining free.

The computer could have been rebooted etc in between. It makes no difference.

In addition, it happens from cold boot and 80%+ of the ram remains free (green) during the entire import. So it doesn't appear to be a "running out of ram" problem.

ronny courtens wrote: If the amount of data to be processed widely exceeds the amount of RAM you have, even when not all available RAM is in use, virtual memory allocation will occur: pages of memory will be written to your system drive. Writing to and reading from a disk is much slower than writing to and reading from your RAM. If the amount of allocated memory is so large in relation to the available RAM you have this can lead to disk thrashing.

Disk thrashing occurs when the system spends a disproportionate amount of time handling page faults and reading and writing pages, rather than executing code for a program,

Paging of any kind, and disk thrashing in particular, affects performance negatively because it forces the system to spend a lot of time reading and writing to disk. Reading a page in from the backing store takes a significant amount of time and is much slower than reading directly from RAM. If the system has to write a page to disk before it can read another page from disk, the performance impact is even worse.

That's why importing from multiple 32 GB cards a the same time will take much longer than importing them one by one.

- Ronny

And if the ram is all still free (green) and it's not being thrashed?

I think I better take this up with Apple....

I think I better take this up with Apple....

Please do, the process I have described was quoted from Apple.

- Ronny

Hi Dave,
In your original post you say you are importing from 5 sd cards, later you say transferring files (disk images) between ssd drives, which is it? Transfers of only 35MB/s suggest either sd cards or maybe via usb2 external drive, not between ssd drives with a fast IO, i.e. USB3/FW800 or Thunderbolt let alone internal PCI cards/bus. Could you describe what the transfer set up is?
What Mac and how much RAM? What is your total set up that is being used.
Activity Monitor will show you pages in/out and as Ronny says see if your Mac is experiencing lots of page swaps.
Finally, have you actually let FCPX transfer the files or are you canceling the process when you see it would take an "estimated" time of 5-6 hours. Macs are notorious for showing much longer estimated time for transfers in the beginning but in reality the process takes only a fraction of the "estimate".
Hope this helps, Greg

VidGreg wrote: Hi Dave,
In your original post you say you are importing from 5 sd cards, later you say transferring files (disk images) between ssd drives, which is it?

My original post doesn't mention SD cards from what I can tell. What I said was:

Example, if I import 5 32GB cards from a Canon XF100, one at a time, they each take around 13 minutes. That's around 65 minutes total.

Canon XF100 uses fast CF cards for 50mbps mpeg2 4:2:2 recording. However, ALL cards had first been copied to the HDD (actually SSD) prior to importing. So I was importing from an SSD drive not from CF or SD cards.

Transfers of only 35MB/s suggest either sd cards or maybe via usb2 external drive, not between ssd drives with a fast IO, i.e. USB3/FW800 or Thunderbolt let alone internal PCI cards/bus. Could you describe what the transfer set up is?

The cards have all been copied to an internal SSD drive that runs > 250MB/s. FCPX is importing from this drive. When importing just one card at a time it goes MUCH faster. When I queue several cards (card images) it slows to a crawl.

What Mac and how much RAM? What is your total set up that is being used.

MacPro 3,1 16GB ram, ATI 5770. Internally 2x SSD drives (connected to the normally hidden SATA connectors) plus 4x2TB HDDs as RAID 0.

Activity Monitor will show you pages in/out and as Ronny says see if your Mac is experiencing lots of page swaps.

Page swaps were insignificant in terms of what was happening. I'm familiar with virtual memory and how page swapping works, but this was not something that was having any significant effect during the import.

Finally, have you actually let FCPX transfer the files or are you canceling the process when you see it would take an "estimated" time of 5-6 hours. Macs are notorious for showing much longer estimated time for transfers in the beginning but in reality the process takes only a fraction of the "estimate".
Hope this helps, Greg

Actually completed the transfer.