Adding timecode to a recorded video

[helgehoi]

Hi and thanks for making this repo.

Would you please provide an example of how to write timecode onto a MOV file with your SDK? And is this possible to do while recording with an AVCaptureSession, or does it have to be added to the file after recording?

Thank you again,
Helge

[orchetect]

Currently timecode tracks can be modified on AVMutableMovie. It's a new feature so there isn't really any documentation quite yet.

I'm working on a demo project that will show how to modify a QuickTime video's timecode track, as well as seek an AVPlayer to timecode.

The feature may become more versatile in future, as time allows. I'm not very familiar with AVCaptureSession but maybe there's potential to add methods for it as well.

For now, this is the method to (over)write a timecode track. Because of limitations of AVFoundation, it may involve saving the video file to disk more than once depending on your workflow.

This is one way to make an AVMovie into a mutable AVMutableMovie if needed:

let movie = AVMovie( ... )
let mutableMovie = movie.mutableCopy() as! AVMutableMovie

Then add/replace the timecode track:

// replace existing timecode track if it exists, otherwise add a new timecode track
try mutableMovie.replaceTimecodeTrack(
    startTimecode: Timecode(TCC(h: 0, m: 59, s: 58, f: 00), at: ._24),
    duration: mutableMovie.durationTimecode(),
    fileType: .mov
)

Finally, the new file can be saved back to disk using AVAssetExportSession. There are other ways of course but this is the vanilla method.

let export = AVAssetExportSession(
    asset: mutableMovie,
    presetName: AVAssetExportPresetPassthrough
)
export.outputFileType = .mov
export.outputURL = // new file URL on disk
export.exportAsynchronously {
    // completion handler
}

[helgehoi]

Thank you for this! If you ever find out a way to add it to the AVCaptureSession directly, that would be awesome.

[orchetect]

I had a quick look through the documentation for AVCaptureSession and its output objects and didn't see anything obvious with regards this. Digging deeper later on may reveal something.

Just further to the above, I realize now you probably want to get the video's duration in order to supply it to replaceTimecodeTrack().

let durationTC = try mutableMovie.durationTimecode()

In a future TimecodeKit release it will be possible to omit the duration parameter to auto-detect the video duration.

[helgehoi]

thx again - that was also very useful

[helgehoi]

Hi! For one of my video files I always get this error-message:

For all others, everything works fine. All files are recorded by the same code on the same phone, so I don't see any reason why this file is different than the rest.

It happens in this code:

try mutableMovie.replaceTimecodeTrack(
startTimecode: myTimeCode,
fileType: .mov
)

[orchetect]

Are you able to log the specific error? That might give more detail on the cause. Error 0 may be pointing to the first case but I'm not sure.

public enum MediaParseError: Error {
    case missingOrNonStandardFrameRate
    case unknownTimecode
    case internalError
    case noData
}

Generally if it fails it is because frame rate could not be detected from the movie file. This will happen usually if the frame rate is non-standard or if the file is missing a video track. But it could be an edge case too.

If you try to read mutableMovie.timecodeFrameRate(), does it return a value or throw an error?

[helgehoi]

Thanks again for helping.

With mutableMovie.timecodeFrameRate() I get the same error on this file, not on the others:

"The operation couldn’t be completed. (TimecodeKit.Timecode.MediaParseError error 0.)"

The strange thing is that ffprobe and QuickTime-player on a mac, as well as Photos on an iPhone, all say that the file has 25 fps.

[orchetect]

This is a bit of a grey area actually.

VFR (variable frame rate) is a way of codecs (like HEVC) compressing/encoding video to be efficient on file size. Generally there should be an effective (or target) frame rate, such as 25 fps. But I'm finding in my research that some formats are a bit unpredictable in this area.

When I feed this video into QuickTime Player and Apple Compressor they treat it as 25 fps.

When I load it into Avid Pro Tools, it sees it as 24.998 fps which is a nonsensical rate. It still allows the video to be used, but it behaves oddly and does not recognize it as standard 25 fps.

The thing is, timecode is a reference for video frames that should technically exist. In VFR rates are non-standard and sometimes interpolated based on how the codec works. So timecode has a loose or somewhat undefined meaning in that context.

As soon as you want to convert or use VFR videos, software usually converts or transcodes them to CFR (constant frame rate) encoding. Dragging the video out of Apple's Photos app on a Mac, for instance. Or exporting from QuickTime / Compressor. That is when timecode becomes meaningful.

As far as making TimecodeKit more compatible with VFR video files...

For example, your test video is min/max 24.000/26.087 with a reported rate of 25.000. That's tricky because two of those three numbers are valid frame rates. Which one do we use? It becomes more complicated when you consider the following:

In my tests, other videos with the same codec shot at "25 fps" have been similar to these specs:

• min/max 6.250/25.000 with a reported rate of 17.188
• min/max 6.250/25.000 with a reported rate of 20.588
• min/max 6.250/25.000 with a reported rate of 24.249

Perhaps the auto-detection heuristic could try using the reported rate first, and failing that it could try using the max rate.

[orchetect]

Just further...

Doing introspection on the actual AVMovie object, I can see that:

- nominalFrameRate: 24.997648
- minFrameDuration: CMTime(value: 23, timescale: 600, flags: [valid], epoch: 0)

We can convert minFrameDuration to a frame rate by 1 / (23/600) which produces 26.08695652174.

Invisor is rounding this to 3 decimal places which is where max 26.087 comes from.

Intuiting that, we can assume that Invisor is showing a reported frame rate of 25.000 only because it is rounding 24.997648 in the same way. But it is not in fact exactly 25.

The min of 24.000 however I have no idea how Invisor is coming up with that number. Nothing in all the video metadata I can access would suggest that number anywhere. However I think we can ignore minimum frame rate any way since reported and max are more relevant.

Full metadata extraction

Tracks:
--------------------------------------------------------------------
▿ 2 elements
  - <AVMovieTrack: 0x60000165d760, trackID = 1, mediaType = vide> #0
    - super: AVAssetTrack
      - super: NSObject
  - <AVMovieTrack: 0x60000165ecc0, trackID = 2, mediaType = soun> #1
    - super: AVAssetTrack
      - super: NSObject

🎬 Video track (vide):
- trackID: 1
- segments: [<AVAssetTrackSegment: 0x600003d05880, source = {{0/600 = 0.000}, {31899/600 = 53.165}}, target = {{0/600 = 0.000}, {31899/600 = 53.165}}>]
- metadata: []
- formatDescriptions: [<CMVideoFormatDescription 0x600001842d00 [0x206b707f8]> {
	mediaType:'vide' 
	mediaSubType:'hvc1' 
	mediaSpecific: {
		codecType: 'hvc1'		dimensions: 3840 x 2160 
	} 
	extensions: {{
    BitsPerComponent = 8;
    CVFieldCount = 1;
    CVImageBufferChromaLocationBottomField = Left;
    CVImageBufferChromaLocationTopField = Left;
    CVImageBufferColorPrimaries = "ITU_R_709_2";
    CVImageBufferTransferFunction = "ITU_R_709_2";
    CVImageBufferYCbCrMatrix = "ITU_R_709_2";
    Depth = 24;
    FormatName = HEVC;
    FullRangeVideo = 0;
    RevisionLevel = 0;
    SampleDescriptionExtensionAtoms =     {
        hvcC = {length = 108, bytes = 0x01016000 0000b000 00000000 96f000fc ... 074401c0 2cbc14c9 };
    };
    SpatialQuality = 512;
    TemporalQuality = 512;
    VerbatimSampleDescription = {length = 224, bytes = 0x000000e0 68766331 00000000 00000001 ... 00010001 00000000 };
    Version = 0;
}}
}]
- availableMetadataFormats: []
- timeRange start: CMTime(value: 0, timescale: 600, flags: [valid], epoch: 0) isValid:🟢
- timeRange duration: CMTime(value: 31899, timescale: 600, flags: [valid], epoch: 0) isValid:🟢
- naturalTimeScale: 600
- nominalFrameRate: 24.997648
- mediaDecodeTimeRange: CMTimeRange(start: __C.CMTime(value: 0, timescale: 600, flags: [valid], epoch: 0), duration: __C.CMTime(value: 31899, timescale: 600, flags: [valid], epoch: 0)) isValid:🟢
- mediaPresentationTimeRange: CMTimeRange(start: __C.CMTime(value: 0, timescale: 600, flags: [valid], epoch: 0), duration: __C.CMTime(value: 31899, timescale: 600, flags: [valid], epoch: 0)) isValid:🟢
- estimatedDataRate: 6330648.0
- minFrameDuration: CMTime(value: 23, timescale: 600, flags: [valid], epoch: 0) isValid:🟢

🔊 Audio track (soun):
- trackID: 2
- segments: [<AVAssetTrackSegment: 0x600003d18cb0, source = {{2112/44100 = 0.048}, {31899/600 = 53.165}}, target = {{0/600 = 0.000}, {31899/600 = 53.165}}>]
- metadata: []
- formatDescriptions: [<CMAudioFormatDescription 0x600002710460 [0x206b707f8]> {
	mediaType:'soun' 
	mediaSubType:'aac ' 
	mediaSpecific: {
		ASBD: {
			mSampleRate: 44100.000000 
			mFormatID: 'aac ' 
			mFormatFlags: 0x0 
			mBytesPerPacket: 0 
			mFramesPerPacket: 1024 
			mBytesPerFrame: 0 
			mChannelsPerFrame: 1 
			mBitsPerChannel: 0 	} 
		cookie: {<CFData 0x600003d38070 [0x206b707f8]>{length = 39, capacity = 39, bytes = 0x03808080220000000480808014401400 ... 1208068080800102}} 
		ACL: {Mono}
		FormatList Array: {
			Index: 0 
			ChannelLayoutTag: 0x640001 
			ASBD: {
			mSampleRate: 44100.000000 
			mFormatID: 'aac ' 
			mFormatFlags: 0x0 
			mBytesPerPacket: 0 
			mFramesPerPacket: 1024 
			mBytesPerFrame: 0 
			mChannelsPerFrame: 1 
			mBitsPerChannel: 0 	}} 
	} 
	extensions: {{
    VerbatimSampleDescription = {length = 143, bytes = 0x0000008f 6d703461 00000000 00000001 ... 00000008 00000000 };
}}
}]
- availableMetadataFormats: []
- timeRange start: CMTime(value: 0, timescale: 600, flags: [valid], epoch: 0) isValid:🟢
- timeRange duration: CMTime(value: 31899, timescale: 600, flags: [valid], epoch: 0) isValid:🟢
- naturalTimeScale: 44100
- nominalFrameRate: 43.066406
- mediaDecodeTimeRange: CMTimeRange(start: __C.CMTime(value: 0, timescale: 44100, flags: [valid], epoch: 0), duration: __C.CMTime(value: 2347008, timescale: 44100, flags: [valid], epoch: 0)) isValid:🟢
- mediaPresentationTimeRange: CMTimeRange(start: __C.CMTime(value: 0, timescale: 44100, flags: [valid], epoch: 0), duration: __C.CMTime(value: 2347008, timescale: 44100, flags: [valid], epoch: 0)) isValid:🟢
- estimatedDataRate: 83125.984
- minFrameDuration: CMTime(value: 0, timescale: 0, flags: [], epoch: 0) isValid:🛑

[orchetect]

TimecodeKit actually already follows the proposed heuristic, but I am making the frame rate detection more forgiving which should allow it to work in a lot more VFR contexts.

[helgehoi]

Thank you. VFR really messes things up. I've found that happens especially when low power mode is enabled on the phone while recording, so I am adding a user warning to turn that off.

For adding the timecode track I provide your code with the nominal frame rate as you suggested in your workaround. That seems to work, although it may not be totally accurate.

I will try to get some large scale data on this as we move forward.

Thanks again for providing the repo!

[orchetect]

It's good to gain insight into these edge cases because it helps improve the library.

TimecodeKit v2 will have the improved frame rate detection. If you're curious you can pull the v2 branch and test it. There will be an official version 2 release this week likely.