fbpx
Welcome, Guest
Username: Password: Remember me
{JFBCLogin}
25 Jan 2021
New boarders will have their posts moderated - Don't worry if you cannot see your post immediately.
Read More...
  • Page:
  • 1

TOPIC:

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 11:43 #122037

Hi all,

As the title suggests, I am having an issue where I cannot get the proxies to link correctly.

I created a new library. Imported the camera originals (mostly FX3, some FX9), leaving the files in place. Once everything had imported, I selected to create proxies which it did (Prores Proxy 1/2 size).

When it had completed transcoding, i went to edit and I noticed the playback was very poor, dropping frames and unuseable. This was in 'Proxy only" mode. When I went to reveal the proxy files in finder, it revealed that the proxies were linked to the original files.

The library and original media and proxies are all located on a QNAP NAS. I have tried moving the proxies to a different drive but it still links to the original after I try to relink. I have trashed prefs, I've made a new library and tried to manually relink to the proxies but to avail.

Before I choose the last resort of just transcoding everything to LT and working from that instead of the proxy workflow, is there anything I'm missing? I think it might be connected with the NAS but I have never had this issue before on this NAS.

Thanks in advance,
R

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 12:12 #122038

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576

....Imported the camera originals (mostly FX3, some FX9), leaving the files in place. Once everything had imported, I selected to create proxies which it did (Prores Proxy 1/2 size).

When it had completed transcoding, i went to edit and I noticed the playback was very poor, dropping frames and unuseable. This was in 'Proxy only" mode. When I went to reveal the proxy files in finder, it revealed that the proxies were linked to the original files....The library and original media and proxies are all located on a QNAP ...

"Reveal in Finder" will only show the original media location. That is normal and doesn't indicate anything wrong. To see where a proxy file is, set the viewer in proxy only, then select ONE file in the event viewer, then do Files>Relink Files>Proxy Media, but do NOT relink it. Just look at the bottom of the dialog in light grey letters. That shows where FCP *thinks* the proxies are. Examine that pathname, see if the files are actually there, if permissions are OK, etc.

Proxy files can be in one of two general places (1) Inside the library itself in the Transcoded Media folder for that event, or (2) If before generating the proxies you defined an external storage location using the library inspector's Storage Locations>Modify Settings>Media dialog, the library's transcoded media folder will contain symlinks that point to proxy files in storage location you previously designated.

As a test, in System Preferences>Security & Privacy>Full Disk Access, grant that to FCP.

Another test would be create a small test library, configure it like the 1st one, import a few files (all located similar to the main library), generate proxies, then inspect are the proxy files in the expected location on the NAS. If you then switch to "proxy only" does FCP see those? If you then do Files>Delete Generated Library Files>Delete Proxy Media, does it delete the few proxy files you just generated?

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 12:14 #122039

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99
A possible solution that may help avoid the issues you're experiencing...

If you follow the workflow shown in this video, you could in effect do a "manual" proxy workflow:



You could transcode to ProRes 422 Proxy at 1/2 size. You then relink between the original camera files and the proxies...

Please Log in to join the conversation.

Last edit: by DaveM.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 13:13 #122040

Thanks both for your responses!

First to Joema, thanks for such a detailed reply! On your point about "Reveal in Finder" , there is a second option which is the one I used - "Reveal proxy media in finder" . This should point to the proxy files but as I described it points to the originals. I have a good understanding of the proxy workflow and where the proxies can be stored, but after trying a few different ways for this project I cannot get it to link the proxies correctly:

-I first tried to relink proxies manually (even though they had been generated by FCPX) - I pointed it to the proxy folder and it still chose the originals folder.
- Next, I moved the proxies to a new folder on the same drive. I created an entirely new library, imported the originals and then tried to link to the proxies that FCPX created which I had moved to a new folder. It still chose the originals
- Next, I moved the proxies to an external drive. Made new library, imported originals, manual link to proxies on ext drive, still linked to originals

In my tests, when I create a small library with 4 of the clips, everything works as it should on the NAS. Could this be anything to do with the fact that there are 4300 clips in the library?

@Dave M, thanks for your reply, I think the second two options above are essentially the same approach, although I could manually create the proxies in compressor instead of FCPX to see if this would work. I'd lose another day if I had to wait for that with no guarantee it will work, so I would probably just transcode the originals to LT instead and forget all about doing a proxy workflow, but at least I would know it will work!

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 13:15 #122041

@Joema,

Forgot to confirm that I already have granted full disk access to FCPX

Thanks,
Richie

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 15:02 #122045

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576

...I first tried to relink proxies manually (even though they had been generated by FCPX) - I pointed it to the proxy folder and it still chose the originals folder...In my tests, when I create a small library with 4 of the clips, everything works as it should on the NAS. Could this be anything to do with the fact that there are 4300 clips in the library?...

I can reproduce the behavior...I'm examining it. It's not related to the NAS, nor the 4,300 clips.

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 15:33 #122046

Interesting, thanks for persisting Joe

I need to press ahead with the edit so I'm going to set the files transcoding into LT and then I can work on them tomorrow but I'd still like to solve the issue so hopefully you find a solution.

Thanks!

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 17:02 #122048

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576

..I'd still like to solve the issue so hopefully you find a solution...

If the FCP command Files>Delete Generated Library Files>Delete Proxy Media was ever issued, even if the proxy files themselves were sequestered in another folder or on the NAS and all proxy thumbnails were red, batch relink to them will thereafter not be possible. However relinking proxy media to the original full-res files will be possible.

The reason is FCP compares the metadata within the library SQL tables for each proxy file to the relink target. That includes things like duration, filename, audio config, timecode format, etc. If those metadata entries don't match the file (OR if they are missing) it will not let you batch relink. For some reason if you select a single file it will allow proxy relink; I don't know if that's a bug or intended.

If the proxy metadata entries are missing, the only ones left are for the original media. Probably for that reason it allows proxy relink to the original full-res files. I don't know if that's a bug or intended.

I'm not saying this is what happened in your case but it reproduces the symptoms you observe. See attached XML I captured when reproducing this.

If you have prior backups of that library (whether FCP auto-backups or file-level backups) it might be possible to load those, then relink to your proxies, then load the latest project XML from your current library. If the state of that library is when the original proxies existed -- even if the thumbnails are all red -- the SQL metadata entries will still exist for those proxy files. That might enable relink to the proxies in the new location.
Attachments:

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 17:26 #122049

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99

Thanks both for your responses!

@Dave M, thanks for your reply, I think the second two options above are essentially the same approach, although I could manually create the proxies in compressor instead of FCPX to see if this would work. I'd lose another day if I had to wait for that with no guarantee it will work, so I would probably just transcode the originals to LT instead and forget all about doing a proxy workflow, but at least I would know it will work!

The workflow described in the Ripple Training video I linked to does not rely on FCP "thinking" that any given set of files are either proxies or originals. What Steve Martin shows is how to relink to a new set of files as "originals". This would be what someone else would do if they had a copy of the media and you sent them an XML file and they had to relink. The workflow bypasses any consideration of proper FCP proxy files as far as FCP is concerned (you don't need to change any media playback options in the View menu of the Viewer).

Basically, you have two different folders, and in each you have files with the same names. The difference is that one set is the true original media and the other set contains transcoded files. Watch the video for more details...

I hope this clarifies things a bit...

Please Log in to join the conversation.

Last edit: by DaveM.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 18:34 #122050

Hi Dave,

You're right, I only watched the first little bit of Steve's video and assumed he was describing a proxy workflow with proxies that are created outside of FCPX and linked later. What he's actually describing, which you have clarified, is like an 'offline edit' workflow which I have used with remote editors for sure.

I think though in this case it would be quite messy really, although I could certainly go down that road as is. Chances are I wouldn't be bothered relinking to the originals with xmls etc, I'd just export the proxies if I think the quality is OK (client certainly won't have an issue with that). It's a shame though, I should simply be able to use the proxy workflow as I have done in the past.

Thanks for your help
Richie

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 18:58 #122051

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99
Richie, yeah I'm just offering more of a workaround. As Steve does in the video, you could edit with the "offline" versions of the media then switch back to the original media before final render/export...

Joe's last post is pretty good at explaining the behavior in FCP. I'd have to test things but I think there are some quirks in how FCP handles things (as Joe alludes to). You might be able to get things to work by relinking as Joe describes. It's not clear and would definitely be easier to confirm in person (which is easy to say)...

Please Log in to join the conversation.

Last edit: by DaveM.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 19:09 #122052

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576
For some reason this apparently no longer works 10.6.4 (at least on Apple Silicon). I just tried the exact procedure from the Steve Martin video. Batch relink won't work because it matches no files. Comparing the clip XML between the original vs HEVC proxy files, it appears there's a mismatch in how the duration metadata is expressed. In the original clip it's expressed as a fraction over 48000 but in the proxy file duration is expressed as a fraction over 24000. The quotient is the same but there must be a math evaluation error. See attached.

I manually edited the XML to change the duration parameter, recreated the clip using that XML, then batch relink from orig. media to proxy (as orig. media) worked.

In the single-file relink case it works OK. Only for batch relink does it fail, but in the real world you must do batch relink. This might be some unintended side affect from adding proxy relink in 10.4.9, maybe combined with Apple Silicon. I haven't tested it on Intel yet. Relink to lower-res HEVC files obviously used to work as shown in Steve Martin's video. That was in March 17, 2020 and 10.4.9 with the proxy relink changes was released on August 25, 2020.
Attachments:

Please Log in to join the conversation.

Last edit: by joema.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 20:01 #122053

Just did a quick test on an Intel 5K iMac running 12.5.1 and FCP 10.6.4.

- ProRes HQ clps imported to library by dragging with Leave Files in Place (don't have any FX6 footage handy)
- Changed Storage Locations->Media settings to a folder named Proxies located alongside the library file. Everything is on the internal SSD.
- Right-clicked in the browser to create ProRes Proxy at 50%
- Chose Proxy Preferred in View and saw the shift to lower res
- Right-clicked in browser to Reveal Proxy Media in Finder and it opened to the proxy files in the newly created Proxies folder.

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 20:09 #122054

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576
I'm deleting much my previous post because I may have been mistaken. I think the issue I attributed to Apple Silicon or FCP versions was more likely a mismatch in the file extension during relink, whether proxy and for original media. In my rush I must have done something different between M1 and Intel machines.

You definitely can have a situation like Richie reported where proxies only relink to original media. That is a sign the file extensions on the proxies are not the same as the originals. If they are the same and the other metadata match, then clips which have never had proxies generated by FCP can be relinked to externally-generated proxies -- as proxies.

Richie the first thing to check is whether the file extensions of your proxies are the same file extension as the original media.

Please Log in to join the conversation.

Last edit: by joema.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 20:39 #122055

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99
Joe, I just did a test with 12 media files, using Steve's "proxies as originals" workflow. I used ProRes 422 LT instead of HEVC for the "proxy" files.

One set of files was ProRes 422 LT and the other set ProRes 422. I created the second set using Compressor, with maintain original file metadata selected and audio set to pass thru.

I am able to relink between each set of files by using :File -> Relink Files -> Original Media...". Switching between each set works for all files.

Current versions of everything on a 2019 Mac Pro...

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 20:44 #122056

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576
Thanks for doing that test, see my previous post which I corrected.

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 30 Aug 2022 20:48 #122057

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99

Thanks for doing that test, see my previous post which I corrected.

Okay... ;-)

Timing of posts can be inopportune, at times... ha...

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 31 Aug 2022 10:12 #122063

Again, Thanks to everyone for such an in depth investigation!

First to say, I am on a 2017 iMac, 4.2 Quad i7, Big Sur 11.6.4, FCPX 10.6.4

The Original media is .mp4, the proxies are .mov. To me, this is as it should be no? My proxies will basically always be .mov and my original could be anything but generally .mxf or .mp4.

Also, can I ask why it works when I test with 3 or 4 clips?

I am currently creating new proxies in Compressor on a 2019 MacPro. If I have created a new library, can I import the originals and link these externally created proxies - or have these original clips been altered by having proxies generated in FCPX already?

Please Log in to join the conversation.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 31 Aug 2022 16:00 #122066

  • joema
  • joema's Avatar
  • Offline
  • Platinum Member
  • Platinum Member
  • Posts: 2303
  • Karma: 27
  • Thank you received: 576

...I am currently creating new proxies in Compressor on a 2019 MacPro. If I have created a new library, can I import the originals and link these externally created proxies - or have these original clips been altered by having proxies generated in FCPX already?

I am retesting some of this today to verify current behavior. As interim guidance I suggest do not transcode thousands of files under the assumption that relink will work. It's better to do a sequence of scaled-up tests, e.g, 10 files, then 100, then 500, etc.

We've been discussing two separate cases:

(1) Relink proxies AS PROXIES -- the relinked files would appear in FCP as proxies, seen when you switch the viewer to "Proxy Only".

(2) Relink proxies AS ORIGINAL MEDIA -- IOW the lower-res proxies would replace the high-res original media after the relink. That is what people did before the 10.4.9 upgrade that allowed proxy relink. In this case you'd do the relink, then edit, then before final grading and export, relink back to high-res originals. You never use View Proxy Only in this case. Obviously you should be highly confident that relink will work, else you're stuck with a low-res timeline.

There were several relink criteria for original media or proxies used as original media, which I list below. I have not recently had time to run a bunch of tests to verify relink constraints for original media or for the new proxy relink case. I'll try to do some of that today but I'm leaving on a trip tomorrow.

The relink behavior can be complicated by the FCP "inode lookup" feature. Namely if the original or proxy file is relocated or renamed anywhere on a given APFS or HFS+ disk volume, FCP will transparently relink that. It does not work across volumes. How it works on an NFS volume I don't know. It might work, might not work or might work differently between SMB and NFS or between various NFS implementations. This implies testing should be done with media and proxies on separate volumes and they should be relocated to a different volume before relink. Otherwise FCP might use inode lookup as a shortcut, and you won't see the true relink behavior.

Note: I just did a quick test to see if proxy relink worked AFTER DELETING existing proxies within FCP. That fails, apparently because the file extension did not match. In this case I was using 50% H264 proxies which had a .mov extension and that initially worked. But after deleting those (from FCP perspective -- the proxy files were external and remained there) it would not relink unless the proxies were renamed to match the original media files, which had a mix of .mp4 and .mts extensions. This illustrates one case where proxy relink can mysteriously begin failing. If the original files are .mov and proxies are .mov it would probably work after deleting the 1st batch of proxies. But if original files are .mp4, etc, then if proxies were ever deleted, the proxy relink would no longer work.

FCP relink criteria:

- Files must have the same audio config (same number of channels), but sample rate can differ.

- Pixel aspect ratio should be the same, but it may relink even if different. This could result in a squeezed or stretched frame.

- Relinked file duration must be the same or longer as the timeline *clip*. Note this is timeline clip not file clip. Absent a timeline you can relink to shorter files. But if used in a timeline the file must be long enough to cover the timeline clip.

- File name including suffix must be the same.

- Codec need not be the same, e.g. you can transcode ProRes originals to 720p H264 and relink those.

- If the clip timecode does not match the original file, it will not relink. The error is "The original file and new file have no shared media range". This is due to metadata in the video header labeled "time code of first frame". In some cases it's possible to re-write the timecode of clips to enable relink by using the EditReady feature Metadata>Set Metadata for All>Clip>Timecode or the utility QtChange64: https://www.videotoolshed.com/handcrafted-timecode-tools/qtchange/

- After relinking to a different resolution file, the viewer may show a window-boxed screen. This is typically a cache issue and can sometimes be resolved by deleting the FCPX cache, which is either stored in the library or outside as defined by the library inspector. The cache is a file bundle named LibraryName.fcpcache.

It is often informative to use the 3rd-party tool Invisor to compare side-by-side the video header metadata from the two files when relink fails: apps.apple.com/us/app/invisor-media-file...or/id442947586?mt=12

However if you cannot determine the location of both files (say via a name search across all drives which might contain it), FCP is likely comparing the metadata characteristics of the initially-imported file to the current relink target. Unfortunately that metadata is stored in SQL tables within the library and there's no FCP command to display that. But you can select that clip in the event browser, do File>Export XML, then in Finder rename it to .txt and display it with Quick Look or Preview. Sometimes it's possible by studying that XML to deduce where the two different versions of the file came from. A good diff tool which can be used on XML is Beyond Compare: https://www.scootersoftware.com

Please Log in to join the conversation.

Last edit: by joema.

Proxy workflow - FCPX is linking to Original media instead of Proxy Media 31 Aug 2022 16:29 #122067

  • DaveM
  • DaveM's Avatar
  • Offline
  • Elite Member
  • Elite Member
  • Posts: 314
  • Karma: 1
  • Thank you received: 99

...I am currently creating new proxies in Compressor on a 2019 MacPro. If I have created a new library, can I import the originals and link these externally created proxies - or have these original clips been altered by having proxies generated in FCPX already?

I am retesting some of this today to verify current behavior. As interim guidance I suggest do not transcode thousands of files under the assumption that relink will work. It's better to do a sequence of scaled-up tests, e.g, 10 files, then 100, then 500, etc.

We've been discussing two separate cases:

(1) Relink proxies AS PROXIES -- the relinked files would appear in FCP as proxies, seen when you switch the viewer to "Proxy Only".

(2) Relink proxies AS ORIGINAL MEDIA -- IOW the lower-res proxies would replace the high-res original media after the relink. That is what people did before the 10.4.9 upgrade that allowed proxy relink. In this case you'd do the relink, then edit, then before final grading and export, relink back to high-res originals. You never use View Proxy Only in this case. Obviously you should be highly confident that relink will work, else you're stuck with a low-res timeline.
...

In addition to Joe's terrific explanation, I would add that EditReady is a wonderful tool, even for seemingly trivial things such as re-wrapping media files. A side-effect of using EditReady is that it "cleans up" media files and associated metadata. For tree-structured original/camera media, it can "flatten" those files and make them more compatible with FCP.

I regularly "launder" camera original media through EditReady, even if it's just to re-wrap files into the .mov container format. This can help avoid many issues, or quirks, inherent in FCP's internal media management. In other words, I try to reduce friction whenever possible...

Of course, as Joe also recommends, workflow testing should be done before committing to anything (ideally, before a project really begins).

By using various prophylactic measures, as far as I can remember, I've never really had a problem with either proxy scenario we've been discussing. Of course, when you add in NAS storage things can get complicated. One reason the Jellyfish is so well-regarded is that it is one of the few network storage devices that is purpose-built (and tested) for use with NLEs.

I've found the content from the first five, or so, results of a search in the FCP User Guide (for "proxy media workflow") quite informative:

support.apple.com/kb/index?page=search&s...proxy+media+workflow

Please Log in to join the conversation.

Last edit: by DaveM.
  • Page:
  • 1