[azservicebus] Better ReceiveAndDelete behavior when receiving then closing the link

[richardpark-msft]

When doing this pattern, with ReceiveAndDelete links, you can lose messages even if you do everything right:

messages, err := receiver.ReceiveMessages(ctx, n > 1)
// this limbo area is a problem
receiver.Close()

The problem here is if there are any excess credits after ReceiveMessages finishes (for instance, if you ask for 100, but only get 3) then those messages will be cached, and lost when Close() is called.

I think we've got two solutions ahead of us, and I've already rejected one of them, making the rest of this process pretty easy:

• REJECTED: Allow the user to call into ReceiveMessges and only receive cached messages. This has some precedent, as you can do this in some of the other clients, but you, as a consumer, wouldn't know how many messages are cached and you still have a race condition between messages arriving and Close().
• CHOSEN: Bring back the 'drain' approach we used, prior to our release callback method. This has the benefit of making sure that every ReceiveMessages() finishes clean - we'll never have any messages in our internal caches and can always make sure the user has everything.

So, we have a few work items for this:

• Bring back the DrainCredits() function, which has been removed in go-amqp. goamqp PR
• Add in drain support into Service Bus
• Update our stress tests to validate that we're not dropping messages in ReceiveAndDelete mode. Probably just take the sample code above, and turn it up to 11.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EldertGrootenboer.

[richardpark-msft]

Status update: DrainCredits has been added, I'm now on the "add and test drain support in Service Bus" portion.