Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

ObjectDisposedException & other exceptions require DeviceClient recreation #1573

Closed
danigian opened this issue Mar 2, 2022 · 1 comment · Fixed by #1574
Closed

ObjectDisposedException & other exceptions require DeviceClient recreation #1573

danigian opened this issue Mar 2, 2022 · 1 comment · Fixed by #1574
Assignees
@atifaziz
Labels
story User Story
Milestone
S02E02 - Discovery 🚀

Comments

@danigian
Copy link
Collaborator

danigian commented Mar 2, 2022 •
edited by p-schuler
Loading

Sometimes C# IoT Hub Device Client is, under the hood, disposing some internal components while we could still need those.

There is some guidance on how to deal with specific state changes #1387

In essence we need to re-create the DeviceClient when any of those happen. The state changes have not shown a conclusive result when connecting to edge. For now we will be handling the exceptions when we see them and then re-create the client. In addition, that will never be a 100% solution and you can immediately run into the same issues, even when you re-connect the client. Therefore, handling the exceptions we see thrown by the client, re-create and retry, is increasing the reliability of the overall system.

#1387 will be something to look into in addition, to be proactive about the re-creation and trying to avoid to run into retry scenarios.

Example stacktrace:

System.ObjectDisposedException:
   at Microsoft.Azure.Devices.Client.AuthenticationWithTokenRefresh+<GetTokenAsync>d__19.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsTokenProvider+<GetTokenAsync>d__3.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.TaskHelpers.EndAsyncResult (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.IteratorAsyncResult`1.StepCallback (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.AsyncResult.End (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.AmqpCbsLink+<>c__DisplayClass4_0.<SendTokenAsync>b__1 (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsLink+<SendTokenAsync>d__2.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpAuthenticationRefresher+<InitLoopAsync>d__10.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotConnection+<CreateRefresherAsync>d__9.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder+<CreateRefresherAsync>d__14.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<EnsureSessionIsOpenAsync>d__28.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<OpenAsync>d__27.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler+<OpenAsync>d__13.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler+<OpenAsync>d__7.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<>c__DisplayClass27_0+<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<ExecuteWithErrorHandlingAsync>d__28`1.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<OpenInternalAsync>d__38.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<EnsureOpenedAsync>d__36.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<>c__DisplayClass29_0+<<SendTwinGetAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<SendTwinGetAsync>d__29.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at TestCode.DeviceClient+<GetTwinAsync>d__8.MoveNext (TestCode.DeviceClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /TestCode/DeviceClient.cs:74)
@danigian danigian added the bug Something isn't working label Mar 2, 2022
@danigian danigian moved this from Triage to In Progress in iotedge-lorawan-starterkit-v2 January 2022 Mar 2, 2022
@atifaziz atifaziz changed the title ObjectDisposedExceptions require a DeviceClient recreation ObjectDisposedException and other exception require DeviceClient recreation Mar 2, 2022
@danigian danigian added this to the S02E02 - Discovery 🚀 milestone Mar 2, 2022
@atifaziz atifaziz changed the title ObjectDisposedException and other exception require DeviceClient recreation ObjectDisposedException & other exceptions require DeviceClient recreation Mar 2, 2022
@atifaziz atifaziz mentioned this issue Mar 2, 2022
Merged
@p-schuler p-schuler moved this from In Progress to In Review in iotedge-lorawan-starterkit-v2 January 2022 Mar 3, 2022
@atifaziz
Copy link
Collaborator

atifaziz commented Mar 3, 2022

Closed via 7256e48 that was merge of PR #1574.

@atifaziz atifaziz closed this as completed Mar 3, 2022
Repository owner moved this from In Review to Done in iotedge-lorawan-starterkit-v2 January 2022 Mar 3, 2022
@atifaziz atifaziz linked a pull request Mar 3, 2022 that will close this issue
Recreate DeviceClient & retry operation on some errors #1574
Merged
@p-schuler p-schuler added story User Story and removed bug Something isn't working labels Mar 4, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@atifaziz atifaziz

Labels
story User Story
Projects
iotedge-lorawan-starterkit-v2 January 2022
Status: Done
Milestone
S02E02 - Discovery 🚀
Development

Successfully merging a pull request may close this issue.

Recreate DeviceClient & retry operation on some errors Azure/iotedge-lorawan-starterkit
3 participants
@atifaziz @danigian @p-schuler