Track position not respecting time scale DSP

[angelobreuer]

Track position not respecting time scale DSP

Transferred from lavalink-devs/Lavalink#521

I was told that my issue regarding the player position would fit more here.

Problem description

Lavalink is a standalone audio sending node based on Lavaplayer and JDA-Audio. Bot developers can play audio using distributed lavalink nodes across different programming languages and bot frameworks.

Lavalink reports a state for each player about every five seconds, containing the current UNIX timestamp when the player update was created and a relative track position. Based on this information, the track's position can be computed. As lavaplayer now has a timescale filter, it is possible to alter the track speed and rate. The position of the track reported by lavaplayer does not respect the actual speed of the track. Calculating the position of the track on the client-side would be possible but would come with a higher error rate because of network latencies, filter jitter (the client does not know the exact time when the filter becomes active), ... (There are some times where the client has to interpolate the position of the track, which corresponds to the time difference between the current host machine time and the time of the last update, which error rate should be ignorable [e.g., errors because of different clock times due to NTP synchronization errors]).

The client state is generated in Lavalink, which you can see at the following: https://github.com/freyacodes/Lavalink/blob/85a54966218094bc06a4f1506003c55523462b04/LavalinkServer/src/main/java/lavalink/server/player/Player.java#L131
There you can see a call to lavaplayer, which requests the current track position.

When using the timescale filter, lavaplayer reports a wrong track position (it uses the position as if the timescale filter has not been applied).

The relation to the actual track position and the reported track position can be observed in the following table:

Position	Time	Delta Position	Delta Position / Time passed since last update
0	1632169708790	0	0
4820	1632169713638	4820	0.964
9820	1632169718631	5000	1
14800	1632169723627	4980	0.996
19800	1632169728629	5000	1
change time scale to 4x
24800	1632169733632	5000	1
29800	1632169738629	5000	1
34820	1632169743629	5020	1.004
39800	1632169748635	4980	0.996
44800	1632169753637	5000	1
change time scale to 2x
49820	1632169758639	5020	1.004
54820	1632169763636	5000	1

Proposal

I would propose that the code for calculating/tracking the track position should be fitted to report a correct track position.

Additional information

@alula commented the following on the original post, which I linked here:

The issue is actually in lavaplayer itself since it's filter framework isn't really meant to change speed of songs. One of solutions would be adding "speed modifier" property to filters and modifying lp's playback time tracking code to take care of that information.

(lavalink-devs/Lavalink#521 (comment))

Alternatives considered

Exposing an additional field that reports the correct time, e.g.:

  /**
   * @return Get the current position of the track in milliseconds
   */
  long getPosition();

  /**
   * @return Get the current position (with respecting time stretching DSPs) of the track in milliseconds
   */
  long getActualPosition();

(from

lavaplayer/main/src/main/java/com/sedmelluq/discord/lavaplayer/track/AudioTrack.java

Line 37 in 84f0d36

long getPosition();

)

Version information

Output of java -version:

java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment Microsoft-18724 (build 11.0.10+9)
OpenJDK 64-Bit Server VM Microsoft-18724 (build 11.0.10+9, mixed mode)

Lavaplayer version (actually a fork):

https://github.com/freyacodes/Lavalink/blob/8759d12f31f8e6b31f4c42eb7a379da68a7d3187/LavalinkServer/build.gradle#L42

---

Original issue created in the Lavalink repository (Click to expand)

Description

Some time ago, Lavalink/Lavaplayer added support for DSP with various embedded effects. When applying the time scale filter, users can control the audio's speed/rate (and pitch, but irrelevant to this issue). For library developers, it is now hard to compute the actual position of the track. The player update that is regularly sent to the client only contains a timestamp (used to synchronize) and a relative position that both represent an actual position in the track. This position is not the actual position on the track. When using the time scale filter, for example, at speed 3, the position given in the player update follows the regular time flow, as no time scale had been applied.

Here is also a dump that shows the position property does not represent the actual position in the track:

Position, Time, Delta Position, Delta Position / Time passed since last update
0, 1632169708790, 0, 0
4820, 1632169713638, 4820, 0.964
9820, 1632169718631, 5000, 1
14800, 1632169723627, 4980, 0.996
19800, 1632169728629, 5000, 1
change time scale to 4x
24800, 1632169733632, 5000, 1
29800, 1632169738629, 5000, 1
34820, 1632169743629, 5020, 1.004
39800, 1632169748635, 4980, 0.996
44800, 1632169753637, 5000, 1
change time scale to 2x
49820, 1632169758639, 5020, 1.004
54820, 1632169763636, 5000, 1

Proposal

I propose adding a property to the player state that represents the current position in the track measured by the samples played. This would help in effectively providing the exact position in the track while respecting the time scale filter. From the client side's view, the track ends prematurely without being even played entirely, as the player update provides the information that the track has started while it is at its end.

An example payload as proposed could like the following:

{
    "op": "playerUpdate",
    "state": {
        "connected": false,
        "position": 5000,
        "actual_position": 14900, // new
        "time": 1632171934331
    },
    "guildId": "redacted"
}

The actual position would represent the samples (converted to milliseconds) played.

The problem for the client is that it would have to keep track of all time scale changes in the track. For example, suppose the user changes the time scale while playing the track. In that case, the tile where the time scale effect is active must be included in the computation and could lead to slight errors while calculating the position in the track (due to network latencies, time used to apply the filter, jitter while playing the track, audio buffering, ...).

Alternatives considered

An alternative would be to either compute the position of the track fully at the client-side.

Another alternative would be to change the position property currently sent with every player update to respect the time scale. But this could be counted as a breaking change, as present clients might not expect this new behavior; the benefit would be that library authors do not have to do very much (except scaling the time passed between the current time and the timestamp).

Version info

** client used:**

Lavalink4NET (currently unreleased version, https://github.com/angelobreuer/Lavalink4NET). Reproducible with a native WebSocket connection.

Output of java -version:

java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment Microsoft-18724 (build 11.0.10+9)
OpenJDK 64-Bit Server VM Microsoft-18724 (build 11.0.10+9, mixed mode)

Version:        8759d12f31f8e6b31f4c42eb7a379da68a7d3187-SNAPSHOT
Build:          1245
Build time:     19.09.2021 10:20:18 UTC
Branch          dev
Commit:         8759d12
Commit time:    19.09.2021 10:17:16 UTC
JVM:            11.0.10
Lavaplayer      1.3.94-original

(Latest CI build)

Thank you very much.

Thank you very much.