LipSync without spectrum file

[drojf]

It might be possible to implement lipsync without using our current method of spectrum files:

Advantages of not using spectrum files:

1. Reduce the number of files on disk - currently there is one spectrum file per each voice file
2. No need to regenerate spectrum files if new voice files are added
3. Since we don't have access to the tool which was used to generate the spectrum files, we'd need to re-write it again (not sure if this has been done already), although it shouldn't be that complicated
4. Can adjust parameters on-the-fly (sensitivity etc) which would allow for easier tweaking

See this unity question for info on doing this..

edit: I did a quick test, and you can get the audio data - I did a quick test which prints every 1000th sample of channel 0, when inserted into the WaitForLoad() function in AudioLaterUnity.cs:

if(audioClip == null)
{
    Debug.Log($"Audio {filename}: audioClip is null!");
}
else
{
    Debug.Log($"Audio {filename}: audioClip is not null! Samples: {audioClip.samples} Channels: {audioClip.channels}");
    float[] samples = new float[audioClip.samples * audioClip.channels];
    audioClip.GetData(samples, 0);
    
    System.Text.StringBuilder sb = new System.Text.StringBuilder();
    sb.Append("[");
    for (int sample = 0; sample < 100*1000; sample += 1000)
    {
        int stride = audioClip.channels;
        int index = sample * stride;
        if (index > samples.Length)
        {
            break;
        }

        sb.Append($", {samples[index]}");
    }
    sb.Append("]");

    Debug.Log($"Audio {filename}: Every 1000 samples (tot 100 samples) CH1: {sb}");
}

[drojf]

While this feature has been added in all chapters except console arcs, there could still be some improvements to the lipsync algorithm.

A new issue has been raised focusing on improving the lipsync algorithm: #89