How to switch the adaptation set (trackGroup) associated with a video renderer on the fly without restarting the player

[Shrihari1428]

Hi Everyone,

I am creating an implementation of exoplayer where I have 4 video renderers (created 3 extra). I have modified my DASH manifest in such a way that each adaption set represents a camera angle and one adaptation set for audio. So now I assign each video adaptation set (video trackGroup) to a video renderer. So if there are 6 video adaptation sets, 4 renderers are initialised with 4 adaptation sets initially. Now I want to switch the adaption set associated with a renderer with an adaptation set thats not in use by specifying the trackGroup id and the renderer index . How do I do this?

Below is the implementation of my CustomTrackSelector

public class CustomTrackSelector extends DefaultTrackSelector {

    public CustomTrackSelector(Context context) {
        super(context);
    }

    @NonNull
    @Override
    protected ExoTrackSelection.@NullableType Definition[] selectAllTracks(
            @NonNull MappedTrackInfo mappedTrackInfo,
            @NonNull @RendererCapabilities.Capabilities int[][][] rendererFormatSupports,
            @NonNull @RendererCapabilities.AdaptiveSupport int[] rendererMixedMimeTypeAdaptationSupports,
            @NonNull Parameters params
    ) throws ExoPlaybackException {
        int rendererCount = mappedTrackInfo.getRendererCount();
        ExoTrackSelection.@NullableType Definition[] definitions =
                new ExoTrackSelection.Definition[rendererCount];
        // Custom change start
        // Get multiple selected videos if renderers available
        @Nullable
        ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectedVideos =
                selectVideoTracks(
                        mappedTrackInfo,
                        rendererFormatSupports,
                        params,
                        (int rendererIndex, TrackGroup group, @RendererCapabilities.Capabilities int[] support) ->
                                VideoTrackInfo.createForTrackGroup(
                                        rendererIndex, group, params, support, rendererMixedMimeTypeAdaptationSupports[rendererIndex]),
                        VideoTrackInfo::compareSelections
                );
        if (selectedVideos != null) {
            for (Pair<ExoTrackSelection.Definition, Integer> selectedVideo: selectedVideos) {
                @Nullable
                Pair<ExoTrackSelection.Definition, Integer> selectedImage =
                        params.isPrioritizeImageOverVideoEnabled || selectedVideo == null
                                ? selectImageTrack(mappedTrackInfo, rendererFormatSupports, params)
                                : null;

                if (selectedImage != null) {
                    definitions[selectedImage.second] = selectedImage.first;
                } else if (selectedVideo != null) {
                    definitions[selectedVideo.second] = selectedVideo.first;
                }
            }
        }
        // Custom change end
        @Nullable
        Pair<ExoTrackSelection.Definition, Integer> selectedAudio =
                selectAudioTrack(
                        mappedTrackInfo,
                        rendererFormatSupports,
                        rendererMixedMimeTypeAdaptationSupports,
                        params);
        if (selectedAudio != null) {
            definitions[selectedAudio.second] = selectedAudio.first;
        }

        @Nullable
        String selectedAudioLanguage =
                selectedAudio == null
                        ? null
                        : selectedAudio.first.group.getFormat(selectedAudio.first.tracks[0]).language;
        @Nullable
        Pair<ExoTrackSelection.Definition, Integer> selectedText =
                selectTextTrack(mappedTrackInfo, rendererFormatSupports, params, selectedAudioLanguage);
        if (selectedText != null) {
            definitions[selectedText.second] = selectedText.first;
        }

        for (int i = 0; i < rendererCount; i++) {
            int trackType = mappedTrackInfo.getRendererType(i);
            if (trackType != C.TRACK_TYPE_VIDEO
                    && trackType != C.TRACK_TYPE_AUDIO
                    && trackType != C.TRACK_TYPE_TEXT
                    && trackType != C.TRACK_TYPE_IMAGE) {
                definitions[i] =
                        selectOtherTrack(
                                trackType, mappedTrackInfo.getTrackGroups(i), rendererFormatSupports[i], params);
            }
        }
        return definitions;
    }

    private <T extends TrackInfo<T>> ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectVideoTracks(
            MappedTrackInfo mappedTrackInfo,
            int[][][] rendererFormatSupports,
            Parameters params,
            TrackInfo.Factory<T> trackInfoFactory,
            Comparator<List<T>> selectionComparator
    ) {
        if (params.audioOffloadPreferences.audioOffloadMode == AUDIO_OFFLOAD_MODE_REQUIRED) {
            return null;
        }
        @C.TrackType int trackType = C.TRACK_TYPE_VIDEO;
        ArrayList<List<T>> possibleSelections;
        ArrayList<ArrayList<List<T>>> possibleSelectionsPerRenderer = new ArrayList<>();
        int rendererCount = mappedTrackInfo.getRendererCount();
        for (int rendererIndex = 0; rendererIndex < rendererCount; rendererIndex++) {
            possibleSelections = new ArrayList<>();
            if (trackType == mappedTrackInfo.getRendererType(rendererIndex)) {
                TrackGroupArray groups = mappedTrackInfo.getTrackGroups(rendererIndex);
                for (int groupIndex = 0; groupIndex < groups.length; groupIndex++) {
                    TrackGroup trackGroup = groups.get(groupIndex);
                    @RendererCapabilities.Capabilities int[] groupSupport = rendererFormatSupports[rendererIndex][groupIndex];
                    List<T> trackInfos = trackInfoFactory.create(rendererIndex, trackGroup, groupSupport);
                    boolean[] usedTrackInSelection = new boolean[trackGroup.length];
                    for (int trackIndex = 0; trackIndex < trackGroup.length; trackIndex++) {
                        T trackInfo = trackInfos.get(trackIndex);
                        @SelectionEligibility int eligibility = trackInfo.getSelectionEligibility();
                        if (usedTrackInSelection[trackIndex] || eligibility == SELECTION_ELIGIBILITY_NO) {
                            continue;
                        }
                        List<T> selection;
                        if (eligibility == SELECTION_ELIGIBILITY_FIXED) {
                            selection = ImmutableList.of(trackInfo);
                        } else {
                            selection = new ArrayList<>();
                            selection.add(trackInfo);
                            for (int i = trackIndex + 1; i < trackGroup.length; i++) {
                                T otherTrackInfo = trackInfos.get(i);
                                if (otherTrackInfo.getSelectionEligibility() == SELECTION_ELIGIBILITY_ADAPTIVE) {
                                    if (trackInfo.isCompatibleForAdaptationWith(otherTrackInfo)) {
                                        selection.add(otherTrackInfo);
                                        usedTrackInSelection[i] = true;
                                    }
                                }
                            }
                        }
                        possibleSelections.add(selection);
                    }
                }
            }
            possibleSelectionsPerRenderer.add(possibleSelections);
        }
        if (possibleSelectionsPerRenderer.isEmpty()) {
            return null;
        }
        ArrayList<Pair<ExoTrackSelection.Definition, Integer>> selectedVideos = new ArrayList<>();
        for (ArrayList<List<T>> selections: possibleSelectionsPerRenderer) {
            if (selections.isEmpty()) continue;
            List<T> bestSelection = max(selections, selectionComparator);
            int[] trackIndices = new int[bestSelection.size()];
            for (int i = 0; i < bestSelection.size(); i++) {
                trackIndices[i] = bestSelection.get(i).trackIndex;
            }
            T firstTrackInfo = bestSelection.get(0);
            selectedVideos.add(Pair.create(
                    new ExoTrackSelection.Definition(firstTrackInfo.trackGroup, trackIndices),
                    firstTrackInfo.rendererIndex)
            );
        }
        return selectedVideos;
    }
}

Below is my playerManager code.

public class PlayerManager {
    private final Context context;
    private ExoPlayer player;
    private final SurfaceViewManager surfaceViewManager;
    private final ArrayList<String> cameraNames = new ArrayList<>();
    private static final String TAG = "HELLO";

    public PlayerManager(Context context, SurfaceViewManager surfaceViewManager) {
        this.context = context;
        this.surfaceViewManager = surfaceViewManager;
    }

    @OptIn(markerClass = UnstableApi.class)
    public ExoPlayer initializePlayer(int numOfViews, Map<String, String> replayData){
        for (Map.Entry<String, String> entry : replayData.entrySet()) {
            cameraNames.add(entry.getKey());
        }

        DashMediaSource dashMediaSource = new DashMediaSource.Factory(new DefaultDataSource.Factory(context))
                .createMediaSource(MediaItem.fromUri(Uri.parse("http://192.168.0.113:8009/replay_emo_goal3_manifest.mpd")));

        player = new ExoPlayer.Builder(context)
                .setRenderersFactory(new CustomRenderersFactory(context, numOfViews))
                .setTrackSelector(new CustomTrackSelector(context))
                .build();
        player.setMediaSource(dashMediaSource);
        player.prepare();
        player.setRepeatMode(Player.REPEAT_MODE_ALL);
        player.setPlayWhenReady(true);
        return player;
    }

    public void switchRendererTrack(){
        logActiveAdaptationSets();
        switchRendererTrackGroup(0, 0);
    }

    @OptIn(markerClass = UnstableApi.class)
    public ArrayList<Renderer> setupRenderers(ExoPlayer player) {
        ArrayList<Renderer> videoRenderers = new ArrayList<>();
        for (int i = 0; i < player.getRendererCount(); i++) {
            if (player.getRendererType(i) == C.TRACK_TYPE_VIDEO) {
                videoRenderers.add(player.getRenderer(i));
            }
        }
        surfaceViewManager.attachRenderersToSurfaces(player, videoRenderers);
        surfaceViewManager.attachLabelsToTextViews(cameraNames);
        return  videoRenderers;
    }

    @OptIn(markerClass = UnstableApi.class)
    public void switchRendererTrackGroup(int rendererIndex, int trackGroupIndex) {
        DefaultTrackSelector trackSelector = (DefaultTrackSelector) player.getTrackSelector();
        if (trackSelector != null) {
            DefaultTrackSelector.Parameters.Builder parametersBuilder = trackSelector.buildUponParameters();
            DefaultTrackSelector.MappedTrackInfo mappedTrackInfo = trackSelector.getCurrentMappedTrackInfo();

            if (mappedTrackInfo == null || rendererIndex >= mappedTrackInfo.getRendererCount()) {
                Log.w(TAG, "Invalid renderer index or track info is unavailable.");
                return;
            }

            TrackGroupArray trackGroups = mappedTrackInfo.getTrackGroups(rendererIndex);

            if (trackGroupIndex < 0 || trackGroupIndex >= trackGroups.length) {
                Log.w(TAG, "Invalid trackGroupIndex: " + trackGroupIndex);
                return;
            }

            parametersBuilder.clearOverridesOfType(rendererIndex)
                    .setSelectionOverride(
                            rendererIndex,
                            trackGroups,
                            new DefaultTrackSelector.SelectionOverride(trackGroupIndex, 0)
                    );

            // Apply the new parameters to the TrackSelector
            trackSelector.setParameters(parametersBuilder.build());

            Log.d(TAG, "Switched to TrackGroup " + trackGroupIndex + " for renderer " + rendererIndex);
        }
    }

    @OptIn(markerClass = UnstableApi.class)
    public void logActiveAdaptationSets() {
        if (player == null) {
            Log.w(TAG, "Player is not initialized.");
            return;
        }

        for (int i = 0; i < player.getRendererCount(); i++) {
            if (player.getRendererType(i) == C.TRACK_TYPE_VIDEO) {
                TrackSelection trackSelection = player.getCurrentTrackSelections().get(i);

                if (trackSelection != null) {
                    TrackGroup trackGroup = trackSelection.getTrackGroup();
                    String adaptationSetInfo = "Renderer " + i + ": Adaptation Set Index " + trackGroup.getFormat(0).id;
                    Log.d(TAG, adaptationSetInfo);
                } else {
                    Log.d(TAG, "Renderer " + i + ": No active adaptation set.");
                }
            }
        }
    }

    public void releasePlayer() {
        if (player != null) {
            player.stop();
            player.release();
            player = null;
        }
    }
}

Currently if there are more video trackgroups than the number of video renderers, all the extra trackgroups are assigned to the firstRenderer alone. ie, when I log mappedTrackInfo.getTrackGroups(rendererIndex) i get

Renderer 1 Support Track groups 1,5 and 6
Renderer 2 Supports Track Group 2
Renderer 3 Supports Track Group 3
Renderer 4 Supports Track Group 4

With the current implementation I am able to switch the trackGroup between 1,5 and 6 for renderer one alone but I am not able to set trackGroup 5 or 6 to renderer 2, 3 and 4.

Is there a way such that all the renderers support all the track groups so that I can switch between them or is there any way I can switch the trackGroup of a renderer though it does not have the trackGroup assigned to it.

Also, setSelectionOverride is deprecated, but when I use addOverride and perform the switch, all my other renderers stop playing. Is there a way to prevent this from happening with addOverride.

Below is my current addOverride implementation

parametersBuilder.clearOverridesOfType(rendererIndex)
                    .addOverride(new TrackSelectionOverride(trackGroups.get(trackGroupIndex), 0));