Weird tempo issues when writing to buffer

[sudgy]

I've been having weird tempo issues with fluidsynth when writing to a buffer with fluid_synth_write_s16. I was wanting to use fluidsynth to write to a buffer and then use that buffer with another library. However, for some midis, the timing seems to be off. Here's code that can reproduce the issue:

#include <fluidsynth.h>

#include <iostream>
#include <fstream>
#include <vector>

const int SAMPLE_LENGTH = 44100;
const int CHANNEL_NUMBER = 2;
const int SECONDS = 30;

std::vector<int16_t> samples(SAMPLE_LENGTH*CHANNEL_NUMBER*SECONDS);

fluid_settings_t* settings;
fluid_synth_t* synth;
fluid_player_t* player;

int main(int argc, char* argv[])
{
    if (argc < 3) {
        std::cout << "Please input a soundfont and midi file.";
        return 1;
    }
    settings = new_fluid_settings();
    synth = new_fluid_synth(settings);
    if (fluid_synth_sfload(synth, argv[1], 1) == FLUID_FAILED) {
        std::cerr << "Unable to open soundfount" << std::endl;
        return 1;
    }
    player = new_fluid_player(synth);
    if (fluid_player_add(player, argv[2]) == FLUID_FAILED) {
        std::cerr << "Unable to open midi" << std::endl;
        return 1;
    }
    if (fluid_player_play(player) == FLUID_FAILED) {
        std::cerr << "Unable to play midi" << std::endl;
        return 1;
    }
    if (fluid_synth_write_s16(synth, SAMPLE_LENGTH*SECONDS, &samples[0], 0, 2, &samples[0], 1, 2) == FLUID_FAILED) {
        std::cerr << "Unable to write buffer" << std::endl;
        return 1;
    }

    // Not what I actually want to do, but it will let you hear it straight from the buffer
    std::ofstream file("sound.raw", std::ios::binary);
    file.write((char*)&samples[0], samples.size() * 2);
    file.close();

    return 0;
}

I have attached a few extra files. 001-Battle01.mid works perfectly. However, 015-Theme04.mid gets messed up. I've converted the output of this program with that midi to an ogg and attached that as sound.ogg.

Here's a few things I've managed to figure out myself:

• When playing songs normally from the command line or using an audio driver, it outputs correctly.
• It definitely seems to be fluidsynth, since I'm not using any other audio drivers or anything. I've also reproduced this problem on Debian Testing (at this time Buster) with g++ and on Windows 10 using mingw64's g++.
• I've tried a few different soundfonts and they all have the same problem.
• It seems to be when less instruments are playing. Some songs are fine until one channel has a solo and then it suddenly speeds up.
• The way the tempo messes up is exactly the same every time.

This bug makes fluidsynth unusable for what I'm trying to do. Any help would be greatly appreciated. Thank you!

prog.zip

EDIT: I forgot to mention I'm using fluidsynth 1.1.6.

[derselbst]

At least it works with the fluidsynth exec:

fluidsynth -F test.wav "/usr/share/sounds/sf2/FluidR3_GM.sf2" 015-Theme04.mid

Have to figure out why though...

[derselbst]

Found the difference: the fluidsynth exec always requests 64 frames from fluid_synth_write_*(). This causes fluid_synth_render_blocks() to always be called requesting one block to render. You however request a bunch of audio frames with one call (which is ok). This however will request a bunch more blocks to be rendered from fluid_synth_render_blocks(). Apparently if doing so, we experience a bug.

[derselbst]

Ok, I think I found the problem and pushed a commit. Not quite sure if this is the right place to fix it, but it seems to work. If you want to try it, clone the timing branch and build it. Once built libfluidsynth.so, make sure you set LD_LIBRARY_PATH to the directory of your freshly build .so, so it doesnt use an old installation in /usr/lib or so.

[sudgy]

It works! Thank you so much!

[derselbst]

Sadly d313db5 doesnt really fix it. Found a midi that still has timing problems and running out of polyphony. Again it sound absolutely perfect rendering it with the fluidsynth exec. Need to investigate further.
Donkey Kong 64 (U) 0000006F 00133CA6.zip

[derselbst]

This bug seems to be design made. fluidsynth pretty much relies on the fact to process all events for one single block and then render whatever we just processed. Processing more blocks means processing more events, esp. noteons. But once a note was fluid_synth_noteon() there is no tracking of where this note was turned on within the current block. One single block is just small enough that you cant hear the difference. You will however hear it pretty soon (depending on the MIDI) when processing a couple of blocks. 6b0c8e7 poorly attempts to fix this. However a proper fix will probably require a bit of rewriting the synth and rvoice implementation, and I'm not willing to do this, sry. For now fa3894c just makes it work, though not in a nice way.

[derselbst]

Apparently fa3894c causes glitchy audio with 015-Theme04.mid and parallel render disabled.

fluid_settings_setint(settings, "synth.parallel-render", 0);
fluid_settings_setint(settings, "synth.threadsafe-api", 0);

Not sure what's wrong. I think there's trouble within rvoice_mixer. Maybe I'll refactor that, but it probably wont fix in 1.1.7.

@sudgy The best what you can do for now is to continiously call fluid_synth_write_*() and always retrieve 64 audio samples from it.

[sudgy]

Why not edit fluid_synth_write_*() to call itself a bunch with 64 samples if it's something different? For instance, if you call it asking for 256 samples, it instead calls it with 64 samples four times in a row. There would maybe be issues with using it to get a non-multiple-of-64 number of samples but it could maybe work for now.

[derselbst]

Still hoping to avoid such a hack...

[sudgy]

I'm just thinking it might be useful to put in 1.1.7. You could actually fix it later. It's just an idea.

[derselbst]

@sudgy I updated the timing branch, feel free to test. It basically makes sure that fluidsynth is always threadsafe, so that events are always enqueued internally, which seems to be pretty much what the current implementation expects.

If you test, make sure you listen to the rendered sound for at least a minute. Previously it started out perfectly and got glitchy after 50s.

[derselbst]

Did not experienced this problem again and merged #193 into master. So we should at least have a workaround for this in 1.1.7 release. Let me know if you're having trouble.

[prof-spock]

There still seems to be some problem with the timing; I have tried version 1.1.10, 1.1.11 and 2.0.1 with
drums-test.zip
The midi file and the flac file generated by fluidsynth are not in sync (but they should).
Any ideas?