Add support for synchronously flush

[jn011]

Hi there,

I have recently tried to integrate with the new library in my serverless function which runs on .NET 6.

In Analytics.NET a call to Flush() would block the current thread and upload all events in the queue for a given client to Segment.

In Analytics-CSharp it seems that flush does not force upload events to segment?
Please checkout my reproduction code below and let me know if my usage is incorrect, in the old client this exact flow works perfectly. For this reason I have continued using Analytics .NET in my project for now.

using Segment.Analytics;
using Segment.Analytics.Utilities;

class SegmentLogger : ISegmentLogger
{
    public void Log(LogLevel logLevel, Exception exception = null, string message = null)
    {
        switch (logLevel)
        {
            case LogLevel.Warning:
            case LogLevel.Information:
            case LogLevel.Debug:
                Console.Out.WriteLine("Message: " + message);
                break;
            case LogLevel.Critical:
            case LogLevel.Trace:
            case LogLevel.Error:
                Console.Error.WriteLine("Exception: " + exception?.StackTrace);
                Console.Error.WriteLine("Message: " + message);
                break;
            case LogLevel.None:
            default:
                break;
        }
    }
}

internal class Program
{
    public static void Main(string[] args)
    {
        Analytics.Logger = new SegmentLogger();

        var analytics = new Analytics(new Configuration("__OMITTED__", storageProvider: new InMemoryStorageProvider()));

        var properties = new Dictionary<string, object?>
        {
            { "TesterProperty", "12345!" },
        };
    

        analytics.Track("dummy_event", properties);

        // Issue, flush does not upload to segment synchronously as expected
        analytics.Flush();
    }
}

Thanks!

[wenxi-zeng]

hey @jn011, that is correct. analytics-csharp does not block and wait for flush to be completed. it is designed to work in a fire-and-forget manner to achieve higher throughputs. by explicitly calling Analytics.Flush() it merely sends a flush signal to the worker thread, and worker thread will do the job according to it's schedule.

you can always write your own plugin to make uploads synchronous, but it's not an easy job and strongly discouraged since it will degrade the performance of your app.

could you please provide more context about your use case of needing a sync flush? I can file a feature request if it feels necessary. thanks!

[jn011]

Hey @wenxi-zeng interesting, thanks for sharing.

For my use case I want flush to be synchronous for processing and emptying an events in the queue. All events should still be queued asynchronously ideally.

This is how my flow looks:

1. Serverless function receives x amount of events from an event stream which need to be transformed and tracked in segment
2. For each of these events they go through a pipeline which calls Analytics.Track 1 to many times.
3. After this pipeline finishes we need to ensure that the serverless function has uploaded all the events to segment, my intention is to force this by calling Analytics.Flush. If we don't ensure that all events are uploaded then the serverless function can exit with unsent items still inside the background queue and lost forever.

Any ideas/thoughts are welcome

[wenxi-zeng]

hey @jn011, thanks for the additional detail. this is a valid use case. probably very common for server use cases. so we will add support to it.

[jn011]

Thanks @wenxi-zeng

Please let me know when this is done. It's the missing piece in order of us to migrate to the new API.

Have you got an extensive list of what's improved from Analytics.NET anywhere?

[wenxi-zeng]

@jn011 we don't have a list yet. but performance-wise, you can take a look at this blog that we benchmarked the old and new android/ios sdks. we do have a plan to write another blog for this c# library. analytics-csharp implements the same architecture as all of the Segment next-gen libraries (kotlin/swift/RN2.0/flutter). in addition, it implements a better event pipeline design that improves the throughputs and scalability, makes queueing and flushing events more streamlined and less prone to concurrency issues.

feature-wise:

• as you might have been noticed, this library is a lot more flexible and extensible than the old one.
• you can write plugins to enrich data, to drop off data, to direct the data to a different server, etc.
• you can customize your storage, httpclient, etc.
• you can write your own flush policy to decide when and how to flush data.
• you can choose your preferred json library (Json.NET or System.Text.Json) by targeting .netstandard 1.3 or 2.0, avoiding to have both json library in your app, reduces the size.
• this library is also cross-platform. it works the same across .NET, Xamarin, and Unity

[MichaelGHSeg]

Closed in error, this is still in progress. Sorry!

[jn011]

Hi @wenxi-zeng Any updates on this?

[edsonjab]

Hi @jn011 we are working into this!

[jn011]

Thanks, any further news? @edsonjab

[MichaelGHSeg]

There's a new release that should address this issue. There is now an option to specify an EventPipelineProvider on the Analytics configuration object. Setting it with a new SyncEventPipeline will use a synchronous flush method. Sorry for the delay in getting this out, we had a couple false starts and distractions.

Let us know how it works for you!