Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Dolby Vision Transcoding and Editing Support #1235

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Add Dolby Vision Transcoding and Editing Support #1235

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
563f4d8
Merge pull request #4 from androidx/main
ybai001 Mar 18, 2024
a5524c0
Add Dolby Vision Transcoding and Editing Support
ybai001 Apr 2, 2024
6acfa0c
Add Dolby Vision Transcoding and Editing Support
ybai001 Apr 2, 2024
7b09e6c
Merge branch 'dlb/dovi-transformer/dev' of https://github.com/DolbyLa…
ybai001 Jun 18, 2024
5eb064d
Add test cases for Dolby Vision transformer support
ybai001 Jul 3, 2024
6b6d66c
Update code based on Google comments
ybai001 Jul 29, 2024
2822919
Update test code based on Google comments
ybai001 Aug 5, 2024
d8321cd
Replace assumeTrue() with @SdkSupress()
ybai001 Aug 6, 2024
fd13345
Remove unused imported class
ybai001 Aug 6, 2024
b8d02d9
Update code again based on latest comments
ybai001 Aug 8, 2024
ffecefb
Fix a bug in HdrEditingTest.java
ybai001 Aug 8, 2024
0f43554
Merge branch 'main' into dlb/dovi-transformer/dev
ybai001 Aug 8, 2024
62e1383
Refine code to match the latest branch
ybai001 Aug 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions ...transformer/src/androidTest/java/androidx/media3/transformer/TransformerEndToEndTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1633,6 +1633,37 @@ public void transcode_withOutputVideoMimeTypeAv1_completesSuccessfully() throws
assertThat(exportResult.videoMimeType).isEqualTo(MimeTypes.VIDEO_AV1);
}

@Test
public void transcode_withOutputVideoMimeTypeDolbyVision_completesSuccessfully() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On which device you have tested this which can produce dolby vision metadata?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OPPO flagship series (e.g OPPO Find X6/X7 pro/ultra), ViVo flagship series (e.g. ViVo X90/pro/ultra, X100/pro/ultra) and XiaoMi flagship series.
I tested it on my Pixel phone with Dolby Vision codec integration.

assumeFormatsSupported(
context,
testId,
/* inputFormat= */ MP4_ASSET.videoFormat,
/* outputFormat= */ MP4_ASSET
.videoFormat
.buildUpon()
.setSampleMimeType(MimeTypes.VIDEO_DOLBY_VISION)
.setCodecs(null)
.build());

MediaItem mediaItem = MediaItem.fromUri(Uri.parse(MP4_ASSET.uri));
EditedMediaItem editedMediaItem = new EditedMediaItem.Builder(mediaItem).build();
Transformer transformer =
new Transformer.Builder(context).setVideoMimeType(MimeTypes.VIDEO_DOLBY_VISION).build();

ExportTestResult exportTestResult =
new TransformerAndroidTestRunner.Builder(context, transformer)
.build()
.run(testId, editedMediaItem);
ExportResult exportResult = exportTestResult.exportResult;

String actualMimeType =
retrieveTrackFormat(context, exportTestResult.filePath, C.TRACK_TYPE_VIDEO).sampleMimeType;
assertThat(actualMimeType).isEqualTo(MimeTypes.VIDEO_DOLBY_VISION);
assertThat(exportResult.durationMs).isGreaterThan(0);
assertThat(exportResult.videoMimeType).isEqualTo(MimeTypes.VIDEO_DOLBY_VISION);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are checking the actual mime type, this check can be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just followed Line 1280 and 1333 in transcode_withOutputVideoMimeTypeAv1_completesSuccessfully() and transcode_withOutputAudioMimeTypeAac_completesSuccessfully() respectively.

}

@Test
public void transcode_withOutputAudioMimeTypeAac_completesSuccessfully() throws Exception {
MediaItem mediaItem = MediaItem.fromUri(Uri.parse(MP3_ASSET.uri));
Expand Down
81 changes: 81 additions & 0 deletions ...aries/transformer/src/androidTest/java/androidx/media3/transformer/mh/HdrEditingTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import androidx.media3.transformer.TransformerAndroidTestRunner;
import androidx.test.core.app.ApplicationProvider;
import androidx.test.ext.junit.runners.AndroidJUnit4;
import androidx.test.filters.SdkSuppress;
import java.util.Objects;
import java.util.concurrent.atomic.AtomicBoolean;
import org.checkerframework.checker.nullness.qual.Nullable;
Expand Down Expand Up @@ -137,6 +138,27 @@ public void export_transmuxHlg10File() throws Exception {
assertThat(actualColorTransfer).isEqualTo(C.COLOR_TRANSFER_HLG);
}

@Test
@SdkSuppress(minSdkVersion = 24)
public void export_transmuxDolbyVisionFile() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think transmuxing test case can also go into MediaItemExportTest as it does not need to run on an actual device.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you give me more hint about it?

  1. I added the Dolby Vision test case here because there are export_transmuxHdr10File() and export_transmuxHlg10File() test cases in this file. I did the similar thing in my test case implementation.
  2. In MediaItemExportTest, I found some transmux test cases but I didn't find corresponding test cases for HLG and HDR10. Do I need to add test case for Dolby Vision in this file?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these tests needs to run on real device because in Robolectric tests, FrameworkMuxer does not write to the actual output file and we are doing validations on the actual output file. So we should keep the test case here only. Thanks!

Context context = ApplicationProvider.getApplicationContext();

Transformer transformer = new Transformer.Builder(context).build();
MediaItem mediaItem = MediaItem.fromUri(Uri.parse(MP4_ASSET_DOLBY_VISION_HDR.uri));

ExportTestResult exportTestResult =
new TransformerAndroidTestRunner.Builder(context, transformer)
.build()
.run(testId, mediaItem);
Format trackFormat =
retrieveTrackFormat(context, exportTestResult.filePath, C.TRACK_TYPE_VIDEO);
@C.ColorTransfer
int actualColorTransfer = trackFormat.colorInfo.colorTransfer;
assertThat(actualColorTransfer).isEqualTo(C.COLOR_TRANSFER_HLG);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check the mime type as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure.

String actualMimeType = trackFormat.sampleMimeType;
assertThat(actualMimeType).isEqualTo(MimeTypes.VIDEO_DOLBY_VISION);
}

@Test
public void exportAndTranscode_hdr10File_whenHdrEditingIsSupported() throws Exception {
Context context = ApplicationProvider.getApplicationContext();
Expand Down Expand Up @@ -401,4 +423,63 @@ public void onFallbackApplied(
throw exception;
}
}

@Test
public void exportAndTranscode_dolbyVisionFile_whenHdrEditingUnsupported_toneMapsOrThrows()
throws Exception {
Context context = ApplicationProvider.getApplicationContext();
Format format = MP4_ASSET_DOLBY_VISION_HDR.videoFormat;
assumeDeviceDoesNotSupportHdrEditing(testId, format);

assumeFormatsSupported(context, testId, /* inputFormat= */ format, /* outputFormat= */ null);

AtomicBoolean isFallbackListenerInvoked = new AtomicBoolean();
AtomicBoolean isToneMappingFallbackApplied = new AtomicBoolean();
Transformer transformer =
new Transformer.Builder(context)
.addListener(
new Transformer.Listener() {
@Override
public void onFallbackApplied(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is deprecated, use the alternative.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still not resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll change the code to use

default void onFallbackApplied(
    Composition composition,
    TransformationRequest originalTransformationRequest,
    TransformationRequest fallbackTransformationRequest) {
  MediaItem mediaItem = composition.sequences.get(0).editedMediaItems.get(0).mediaItem;
  onFallbackApplied(mediaItem, originalTransformationRequest, fallbackTransformationRequest);
}

Composition composition,
TransformationRequest originalTransformationRequest,
TransformationRequest fallbackTransformationRequest) {
isFallbackListenerInvoked.set(true);
assertThat(originalTransformationRequest.hdrMode).isEqualTo(HDR_MODE_KEEP_HDR);
isToneMappingFallbackApplied.set(
fallbackTransformationRequest.hdrMode
== HDR_MODE_TONE_MAP_HDR_TO_SDR_USING_OPEN_GL);
}
})
.build();
MediaItem mediaItem = MediaItem.fromUri(Uri.parse(MP4_ASSET_DOLBY_VISION_HDR.uri));
EditedMediaItem editedMediaItem =
new EditedMediaItem.Builder(mediaItem).setEffects(FORCE_TRANSCODE_VIDEO_EFFECTS).build();

try {
ExportTestResult exportTestResult =
new TransformerAndroidTestRunner.Builder(context, transformer)
.build()
.run(testId, editedMediaItem);
assertThat(isToneMappingFallbackApplied.get()).isTrue();
@C.ColorTransfer
int actualColorTransfer =
retrieveTrackFormat(context, exportTestResult.filePath, C.TRACK_TYPE_VIDEO)
.colorInfo
.colorTransfer;
assertThat(actualColorTransfer).isEqualTo(C.COLOR_TRANSFER_SDR);
} catch (ExportException exception) {
if (exception.getCause() != null) {
@Nullable String message = exception.getCause().getMessage();
if (message != null
&& (Objects.equals(message, "Decoding HDR is not supported on this device.")
|| message.contains(
"OpenGL ES 3.0 context support is required for HDR input or output.")
|| Objects.equals(message, "Device lacks YUV extension support."))) {
return;
}
}
throw exception;
}
}
}
4 changes: 4 additions & 0 deletions libraries/transformer/src/main/java/androidx/media3/transformer/DefaultDecoderFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
import com.google.errorprone.annotations.CanIgnoreReturnValue;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;

/**
* Default implementation of {@link Codec.DecoderFactory} that uses {@link MediaCodec} for decoding.
Expand Down Expand Up @@ -229,6 +230,9 @@ public DefaultCodec createForVideoDecoding(
if (SDK_INT >= 31 && requestSdrToneMapping) {
mediaFormat.setInteger(
MediaFormat.KEY_COLOR_TRANSFER_REQUEST, MediaFormat.COLOR_TRANSFER_SDR_VIDEO);
} else if (SDK_INT >= 31 && Objects.equals(format.sampleMimeType, MimeTypes.VIDEO_DOLBY_VISION)) {
mediaFormat.setInteger(
MediaFormat.KEY_COLOR_TRANSFER_REQUEST, MediaFormat.COLOR_TRANSFER_HLG);
}

@Nullable
Expand Down
7 changes: 7 additions & 0 deletions libraries/transformer/src/main/java/androidx/media3/transformer/EncoderUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,13 @@ public static ImmutableList<Integer> getCodecProfilesForHdrFormat(
return ImmutableList.of(MediaCodecInfo.CodecProfileLevel.AV1ProfileMain10HDR10);
}
break;
case MimeTypes.VIDEO_DOLBY_VISION:
if (colorTransfer == C.COLOR_TRANSFER_HLG) {
return ImmutableList.of(
MediaCodecInfo.CodecProfileLevel.DolbyVisionProfileDvheSt);
Copy link
Contributor

@SheenaChhabra SheenaChhabra Jul 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of all profile, why have we chosen only this one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. All exist devices which support Dolby Vision encoding on market will encode the input stream to this profile only. In future, maybe we will extend this list. But Dolby has not finalized which profiles we should support for encoding.

}
// CodecProfileLevel does not support PQ for Dolby Vision.
break;
default:
break;
}
Expand Down
71 changes: 71 additions & 0 deletions libraries/transformer/src/main/java/androidx/media3/transformer/FrameworkMuxer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@

import android.annotation.SuppressLint;
import android.media.MediaCodec.BufferInfo;
import android.media.MediaCodecInfo;
import android.media.MediaFormat;
import android.media.MediaMuxer;
import androidx.annotation.Nullable;
import androidx.annotation.RequiresApi;
import androidx.media3.common.C;
import androidx.media3.common.Format;
import androidx.media3.common.Metadata;
Expand Down Expand Up @@ -122,6 +124,10 @@ public TrackToken addTrack(Format format) throws MuxerException {
if (isVideo) {
mediaFormat = MediaFormat.createVideoFormat(sampleMimeType, format.width, format.height);
MediaFormatUtil.maybeSetColorInfo(mediaFormat, format.colorInfo);
if (sampleMimeType.equals(MimeTypes.VIDEO_DOLBY_VISION) && SDK_INT >= 33) {
mediaFormat.setInteger(MediaFormat.KEY_PROFILE, getDvProfile(format));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please share some corresponding spec which explains these changes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as Dolby Vision and Profile, the spec is here: https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile?language=en_US

And I'd like to add some explanation why I did this code change:

In Media3 library code FrameworkerMuxer.java, addTrack(Format format) method, it calls system MediaMuxer.addTrack(mediaFormat) method without PROFILE/LEVEL settings.

This is OK for other video formats but results in issue for Dolby Vision. The key point is in Android AOSP utils.cpp file, convertMessageToMetadata() method, KEY_PROFILE is checked. If there is no this parameter, BAD_VALUE will be returned.

see: https://cs.android.com/android/platform/superproject/main/+/main:frameworks/av/media/libstagefright/Utils.cpp;drc=216a97bdad3c0f492094250b33d82abf27098932;l=2115

mediaFormat.setInteger(MediaFormat.KEY_LEVEL, getDvLevel(format));
}
try {
mediaMuxer.setOrientationHint(format.rotationDegrees);
} catch (RuntimeException e) {
Expand Down Expand Up @@ -292,12 +298,77 @@ private static ImmutableList<String> getSupportedVideoSampleMimeTypes() {
if (SDK_INT >= 24) {
supportedMimeTypes.add(MimeTypes.VIDEO_H265);
}
if (SDK_INT >= 33) {
supportedMimeTypes.add(MimeTypes.VIDEO_DOLBY_VISION);
}
if (SDK_INT >= 34) {
supportedMimeTypes.add(MimeTypes.VIDEO_AV1);
}
return supportedMimeTypes.build();
}

/**
* Get Dolby Vision profile
*
* <p>Refer to <a
* href="https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile">What
* are Dolby Vision profiles and levels</a>.
*/
@RequiresApi(33)
private static int getDvProfile(Format format) {
// Currently, only profile 8 is supported for encoding
// TODO: set profile ID based on format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the work involved for this TODO?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, only profile 8 (H.265 based) is supported for encoding. In future, maybe Dolby will support other Dolby Vision profile (e.g. profile 9/10). It's not finalized. If that is true, we need to add logic here to decide which profile should be used based on the parameter "format".

return MediaCodecInfo.CodecProfileLevel.DolbyVisionProfileDvheSt;
}

/**
* Get Dolby Vision level
*
* <p>Refer to <a
* href="https://professionalsupport.dolby.com/s/article/What-is-Dolby-Vision-Profile">What
* are Dolby Vision profiles and levels</a>.
*/
@RequiresApi(33)
private static int getDvLevel(Format format) {
int level = -1;
int maxWidthHeight = Math.max(format.width, format.height);
float pps = format.width * format.height * format.frameRate;

if (maxWidthHeight <= 1280) {
if (pps <= 22118400) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelHd24; // Level 01
} else { // pps <= 27648000
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelHd30; // Level 02
}
} else if (maxWidthHeight <= 1920 && pps <= 49766400) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelFhd24; // Level 03
} else if (maxWidthHeight <= 2560 && pps <= 62208000) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelFhd30; // Level 04
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why the value of these constants is not same as mentioned level? For example DolbyVisionLevelFhd30 = 8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the codec profile/level constants should be a bitmask, which is defined by Android AOSP. Dolby defined our profile/level to follow this rule. See https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/media/java/android/media/MediaCodecInfo.java;drc=6a4bef0f90e822e19866e53a98b85029bff04ea0;l=4301

I think Google defined it in this way due to efficiency. For example, to judge whether one profile is supported by current device, you just need to do one time comparison:
if (appointedLevel & (DolbyVisionLevelFhd24 | DolbyVisionLevelFhd24) {
// Current device support this level
} else {
// Current device doesn't support this level
}

} else if (maxWidthHeight <= 3840) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use this existing method?

https://source.corp.google.com/piper///depot/google3/third_party/java_src/android_libs/media/libraries/exoplayer/src/main/java/androidx/media3/exoplayer/mediacodec/MediaCodecUtil.java;l=307?q=MediaCodecUti#:~:text=306-,307,-308

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's for different purpose. The method in your link is to PARSE the Dolby Vision profile and level from codec string. The use case is to play a streaming content. You can get codec string from a MPEG-DASH manifest file. Then paring this codec string, you will get profile/level info. This info can be used to select which Dolby Vision decoder to handle this streaming content (H.264 based Dolby Vision decoder or H.265 based Dolby Vision decoder). In this case, Format.codecs is a valid codec string.

As far as this code contribution case, it is used to decide which Dolby Vision profile/level should be used based on incoming content width/height/framerate. In this case. Format.codecs is NULL.

if (pps <= 124416000) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelFhd60; // Level 05
} else if (pps <= 199065600) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelUhd24; // Level 06
} else if (pps <= 248832000) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelUhd30; // Level 07
} else if (pps <= 398131200) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelUhd48; // Level 08
} else if (pps <= 497664000) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelUhd60; // Level 09
} else { // pps <= 995328000
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevelUhd120; // Level 10
}
} else if (maxWidthHeight <= 7680) {
if (pps <= 995328000) {
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevel8k30; // Level 11
} else { // pps <= 1990656000
level = MediaCodecInfo.CodecProfileLevel.DolbyVisionLevel8k60; // Level 12
}
}

return level;
}

private static class TrackTokenImpl implements TrackToken {
public final int trackIndex;

Expand Down