Refine multi-track support + track selection

[IanDBird]

At the moment, our application will start playing a video (e.g. an MKV via the WebmExtractor) and wait until the onPlayerStateChanged callback is invoked with a playbackState of ExoPlayer.STATE_READY. This event appeared to be the correct time when the extractor had parsed the media file, and therefore knew which tracks were available. It was also before playback had begun, so didn't introduce any user visible issues where the wrong track would temporarily be played.

This works well, except for the scenario when the first track (which is the default one selected) is not actually supported but the application is actually wanting to select a different one. The specific scenario is as follows:

• An Android device that handles DTS but not DTS-HD
• An MKV which contains two audio tracks, 1) DTS-HD 2) DTS
• The application wants to select the DTS track

At the moment, what actually happens is that the player detects that DTS-HD isn't supported and calls the onPlayerError before the onPlayerStateChanged.

My question is when should be trying to select tracks:

• Are there any other events which occur after tracks have been parsed, but before playback is attempted?
• Should/can we disable all audio tracks before playback, ensuring that we then select the one we want?
• Should we detect the onPlayerError was called before we have selected the track and see if we can select it and try again?

Thanks in advance!

[ojw28]

Track selection is pretty complicated. The current approach we take has quite a lot of niggles:

• Filtering of tracks by device capabilities doesn't happen in the right place. It should happen in the renderer, since the renderer is the thing that knows what it's capable of playing. Right now filtering happens way upstream for DASH/SS/HLS and not at all (beyond checking that a decoder exists for the mime type) when using ExtractorSampleSource.
• For DASH/SS/HLS the application doesn't have a nice way to select an adaptive track containing a subset of tracks that match the device capabilities. This would be a good thing to support.
• We don't expose tracks that aren't supported to the application. This would obviously be nice so that the application knows what happened. Right now a video playback where we only support the audio stream looks identical to an audio playback to the application.
• The player makes an initial track selection and the application can only change it asynchronously (i.e. via a process that involves messages passing between threads). This leads to inefficiency in the case that the application wants to select a different track. It also causes problems if the initial track selection for some reason doesn't work (and the application knows this somehow), as you mention above.

We do have plans to fix all of this, although it's quite a lot of work. I'm envisaging something like:

• DASH/SS/HLS components will not filter tracks based on device capabilities or generate an adaptive track. They'll just list all individual tracks. Tracks will be marked with a group if they can be used with other tracks in an adaptive way.
• TrackRenderers will mark tracks as playable or not based on their capabilities.
• All tracks will be exposed through ExoPlayer's API, including those not playable for information purposes.
• We'll allow synchronous track selection on the player thread by letting the application inject some kind of track selector object. This is complicated, because it needs to consider all renderers at once to be most useful, rather than having a separate selector for each renderer separately. This is because in practice dependencies might exist. For example an application might want to enable a subtitle track only if an audio track doesn't exist for a certain language. Note that the injected track selector object will also be of use for things like full multi-period support in DASH where formats can change moving from one period to the next. In such cases I'd envisage the selector that has been injected being invoked again for the next period.

[ojw28]

• The injected track selector would be able to select multiple tracks belonging to the same group to enable adaptive playback, if the renderer says that it supports it.

[IanDBird]

• We don't expose tracks that aren't supported to the application. This would obviously be nice so that the application knows what happened. Right now a video playback where we only support the audio stream looks identical to an audio playback to the application.

I've raised this before, but basically our application ends up having to have a compatibility matrix representing what the current version of ExoPlayer it's using actually supports (e.g. what codecs the extractors can handle). This is to allow translation of the original source files track indexes into ExoPlayer's. It makes it very important that in our application we ensure we update that matrix whenever we update ExoPlayer, otherwise we'll end up with incorrect track selections. What you're suggesting sounds great, and would remove a lot of this complexity.

• We'll allow synchronous track selection on the player thread by letting the application inject some kind of track selector object.

That sounds exactly what we need :) Is there any existing GHI that covers that work that I can follow? Or would it be okay to change this GHI for that change, or even create a new one?

[ojw28]

This is fixed in the 2.x.x experimental branch. You can implement a custom TrackSelector to perform arbitrary selection of tracks for each renderer.

Note that for nearly all use cases, using DefaultTrackSelector and just implementing a custom TrackSelectionPolicy should be sufficient.