TOPIC:

Good news. I edited the XML adding .mov to each of the problem id clips and achieved successful import.

There was just a dozen or so on this first proj TL. The pattern thus far is that all the problem ids are from that "C-CAM" series of clips, but not all "C-CAM" clips used in the show are problems. Tracing those clips from the proj TL > browser > finder I'm seeing file extensions as .mov however on those files.

Not sure the impact across all proj TLs - there are six one-hour proj TLs in total. I worried the workflow. I'm also not sure the best way to collaborate moving forward.

I'm going to update the other board too.

Thanks
Kevyn

DaveM wrote:

... how a shared set of media, where both people were working on the same Project, ended up in this situation. In other words, why would this happen in the first place or why would exporting an XML create this situation (provided the referenced file(s) was already in use in FCP)?...

Exactly. How did this happen? I can reproduce it by deleting the file extension, then exporting XML. But so what? The media files should have gone red and quit working before loading the XML on another machine.

This situation requires backtracking to the original machine where those files are, examining the current names, the names of the symlinks pointing to them, resolving the symlinks using ls -l, which will show the link target whether the final file exists or not.

Possibly something happened on that machine such as files were on a NAS drive, somehow the file extensions were suppressed but kept working then the XML was malformed. Maybe the drive is ExFAT or maybe it's NTFS running Paragon. We need to know:

1. What kind of machine and storage area the source files on.
2. Are those files still working in FCP on that machine.
3. Are the media files and symlinks normal for the problematic files?
4. If they are not normal, how is it still working? They should be red missing media clips.

joema wrote:

... Exactly. How did this happen? ...

Joe,

Yup, I've been thinking that from the start. I asked for some relevant info but it hasn't been forthcoming yet. Anyway, Guessing about the details is not productive. I have some thoughts but I won't share them here without more info.

Here's an overview of the production and editing enviro / situation. The editor is in Southeast Asia and information doesn't move as quickly because of the clock.

This is a doc production of 12+ years covering the story. It resides now as 6 1-hour episodes. There is 80 hours+ of raw footage. And ~2TB of acquired archival. Moving drives and posting at dropboxes (and the like) is problematic. The initial string out was performed by me (Enviro 1 = E1) and media drives and the library duplicated and brought to the editor (Enviro 2 = E2). During the last 12-15 months, additional footage shot, archival accumulated, etc., was added to both E1 and E2 however as separate operations.

1. E1 shows all file extensions
2. Drives are MAC OS Extended (Journaled)
3. Footage left in locations, ingested as proxy
4. FCP 10.6.1 on both machines
5. MAC M1, Monterry 12.6 on both machines

Additionally,
The "C-CAM" series of clips appearing to be the problem are from a lower-end camera, Panasonic HDC-TM700. It uses AVCHD files. On E1 - When I follow them back from Proj to drive, I see the extensions. However on E2 - i'm now seeing that some files don't have the .mov. See attachment.

thank you-

DC-CD wrote:

...The "C-CAM" series of clips appearing to be the problem are from a lower-end camera, Panasonic HDC-TM700. It uses AVCHD files. On E1 - When I follow them back from Proj to drive, I see the extensions. However on E2 - i'm now seeing that some files don't have the .mov. See attachment...

Thanks for the helpful info and the work it took to obtain it. What you show is definitely related to the problem. If The E2 machine has no file extensions for some AVCHD files, that is likely why the exported project XML had no file extensions for those files. On the import side, the FCP XML parser knows that a media file with no extension is invalid, hence the error.

The question is why do those particular files have no file extensions? Normally AVCHD media files have a .MTS file extension, and are inside a package or file bundle. The fact that some of your AVCHD media files have a .MOV extension implies they were transcoded before (or as part of) import to FCP.

If imported to FCP using "copy to library", AVCHD media files often have names like "Clip #1.mov", "Clip #2.mov", etc. The working AVCHD files with .mov file extensions have more meaningful names. This implies they were renamed, which raises the possibility of a renaming error causing some of them to lack file extensions.

There is a Finder advanced setting for "Show warning before changing extension". If that was disabled it would be very easy to rename media files lacking extensions.

If there is a technical or procedural cause of this, it would help the FCP community to figure this out. You are not the only person to encounter this.

My team has a dedicated DIT to rename files, and infrequently even he has made errors with complex RED R3D files.

It does seem like the problem C-CAMs are from one production batch in 2015. And that's the only batch synch'd using plural eyes....hmmm. So far anyways.

I do know my workflow for the AVCHD is to import into to FCP and get the footage files to perform backup. Reading on a MAC is only possible through FCP.

I'll continue to work this problem and report back. There's going to be a lot of back and forth, E1 to E2 again next week and early June through Aug. thanks-

Joe (and Kevyn), my last comments on this issue...

The camcorder that Kevyn mentioned (thanks for the info, Kevyn) uses a media structure with (likely) DCIM, MISC, and PRIVATE folders/directories on the SD card (FAT32, ExFAT, etc.). The PRIVATE folder would (likely) contain an AVCHD folder, with BDMV and STREAM folders inside (amongst others). I would imagine that the actual video files are .MTS files.

So, it seems like the SD card was processed and the video was re-wrapped or transcoded to get the .mov files. How the transfer was done is of interest.

It would be useful to see if there is any difference in the metadata of one of the problem files and one of the files from the same source that has the .mov extension.

I believe that once imported into FCP, video files may also be referenced (internally, by the SQLite DB) by their inodes rather than the just filename, which could expain why removing a filename extension wouldn't affect the file's use in the current FCP Library after import (and not yield a "missing media" warning). However, on export to XML, the actual filename must be used in the filename URL schema.

[Side note: I wonder if how media is referenced in FCP's DB is why drive formats other than HFS+ and APFS are not recommended/supported.]

Are the missing filename extensions a result of filename changes after the media was imported and used initially, or was it something that happened when media was copied from drive to drive? Were the problem files imported at a different time than the good ones (with .mov extensions)? Was the method of import into FCP different (via direct import of the video files, or using the SD card structure, or from a camera card archive)? This issue seems to be more of a "DIT" or assistant editor media management issue than anything to do with FCP specifically.


Cheers.

DaveM wrote:

...I believe that once imported into FCP, video files may also be referenced (internally, by the SQLite DB) by their inodes rather than the just filename, which could expain why removing a filename extension wouldn't affect the file's use in the current FCP Library after import (and not yield a "missing media" warning). However, on export to XML, the actual filename must be used in the filename URL schema...

I also suspected that, but I tested it and it doesn't work. If only the filename is changed (not the extension), then inode lookup will keep it working. But if the extension is changed, then that doesn't work.

DaveM wrote:

.[Side note: I wonder if how media is referenced in FCP's DB is why drive formats other than HFS+ and APFS are not recommended/supported.]

I think there might be some reference (maybe an old one) saying that ExFAT is supported for FCP. However we know from experience it's not a good idea. ExFAT is missing numerous features that HFS+ and APFS have: write-ahead logging for filesystem metadata changes, inode lookup, symbolic links, hard links, etc.

I have never investigated how FCP reacts in situations that depend on those features. IMO it's safer to just not use ExFAT.

DaveM wrote:

...Are the missing filename extensions a result of filename changes after the media was imported and used initially, or was it something that happened when media was copied from drive to drive?...

That is a good question. I think DC-CD's last post about Plural Eyes being used only on the "bad" batch of files is interesting. That syncs the files external to FCP, then generates an XML with the time offsets to generate a synced project. You import that XML and they appear as both a synced project and synced multicam in separate events. I don't remember Plural Eyes changing file extensions but maybe if it generated an old version of fcpxml that might cause some issue.

Still working great on the first project XML. Second XML project TL came over and had a few similar "id" errors which i fixed however while going down the line I hit this new error. (see attached)

It doesn't have an id like the other errors. I can't locate on the XML what is being flagged.
Kevyn

Did you try scanning the XML with that shell script I wrote to see if it finds any media files lacking a proper file extension? If it is too difficult to use, let me know and I'll convert it to an automator script.

Actually I should probably first examine your XML (before you edit it) to see if I can manually spot the problem. If it is a different problem than a media file lacking a file extension, I could also add a check for that to the script.

You can upload your 2nd project XML to me at this secure write-only location: www.dropbox.com/request/w5Tb4uisD75ou49ilWYt

Hi Joema -
It is too difficult. I have no experience with this operation.
Posting file there now. Thanks, Kevyn-

DC-CD wrote:

Hi Joema -
It is too difficult. I have no experience with this operation.
Posting file there now. Thanks, Kevyn-

OK here is a link to a Mac automator script packaged as an .app file:

www.dropbox.com/s/3kt1x5i0177j5pa/find_b...a_names.app.zip?dl=0

Because it's not code signed, it's necessary to open it via right-click and pick "Open", or "Open Anyway". Just disregard the security warning. The .app file does only what the above .sh script does, except I added to disregard .pdf and a couple other file extensions to avoid false positives.

It prompts you for the name of the fcpxml file to check using a standard file/open dialog. Then it asks for what output file to write the results to. Give it a filename ending in .txt, then examine that in Finder, TextEdit, etc.

I ran it on the scripts you sent me and it found a few things. Each of them were like before -- media files with no file extension.

The output file includes line numbers of the source file where it finds problems. You can go to those line numbers in the .fcpxml source file by using any text editor that supports line numbers. With TextEdit you can do CMD+L to enter a line number.

To edit or view a .fcpxml file, in Finder you can right+click and pick "Open in TextEdit".

For editing things like this I use BBEdit.

Please let me know if any problems.

Joema, thanks for the easy to follow instructions. I had to wait to backup my computer before I tried this operation. I easily located and corrected ~17 file names. I'm still getting this other error.

I don't understand how to locate the problem.
Thank you
Kevin

Send me your XML for that project and I'll examine it.

Edit/update: also export and send me the library XML. It's possible there is an erroneous zero frame duration field either in the project or some clip in some event of the library that the project is referencing.

Please ZIP those before uploading to ensure they don't get changed in transit.

Hi Joema
Send to your dropbox now. The Ep 2 XML is the same reviewed earlier. The version i'm posting now has the ~17f fixed file extensions, and I'm posting the report.
Kevyn

Kevyn, I got the files. I'll look at them tonight.

Thank you-

Kevyn, I managed to edit the project so it seems to load. I had to delete a few items which were interdependent. Below is a link to the edited XML, also a Beyond Compare "diff" report of the lines I deleted. Try it out and see if it works.

It may be possible to do fewer edits and fix it, but this is the first thing I found which seemed to work. My procedure was edit and search the XML using BBEdit, which can display on a single screen all matching lines for a search pattern, or display specific matched fields using a grep search pattern.

The error when loading the XML referenced "format[67]". That could be interpreted two ways (1) Something with an ID code of 67 or the 67th ordinal reference in the file for a format tag.

Using the 2nd approach I deleted that line which then produced other errors on loading the XML. To make checking the XML faster I used the MacOS built-in command-line tool "xmllint". This allows checking the XML for errors and also validates it against the Apple-provided DTD file for fcpxml 1.9. The syntax I used was:

xmllint --noout --dtdvalid /Applications/Motion_5_6_3.app/Contents/Frameworks/Interchange.framework/Versions/A/Resources/FCPXMLv1_9.dtd /Users/josephmarler/Downloads/FCPSupportFiles-3/Episode\ 2.9\ louisOrig.fcpxml

With each deleted line or group of lines, there were other lines dependent on those deleted. This in turn caused other errors. So I observed those errors and walked down the chain, each time deleting the lines that complained about the lines I previously deleted. After just a few passes, the errors from xmllint stopped. I loaded the XML in FCP and it seemed to work.

Edited project XML: www.dropbox.com/s/n7jg71z9y796pfo/Episod...ited.fcpxml.zip?dl=0

Beyond Compare "diff" output showing what I deleted: www.dropbox.com/s/ay0hkq51lg5xw5o/Episod...leted_Lines.jpg?dl=0

Hey Joe, would you mind just posting a text file containing the lines you removed. In the screenshot from Beyond Compare some of the ends of lines are out of view. I have a guess as to what is going on, but I'd need to see the complete original file or the full extent of what was removed. Thanks.

For Kevyn: is this new media that you've recently added (that is, not part of the original drive that was copied for you and the editor)?

DaveM wrote:

Hey Joe, would you mind just posting a text file containing the lines you removed...

Dave, here is a .txt file showing just the lines deleted. They are in two contiguous blocks. Before each block I put in parenthesis the line numbers from project XML "Episode 2.9 louis.fcpxml".

www.dropbox.com/s/crno33w2pgh0ye7/Episod...nes_deleted.txt?dl=0