Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java #365

mssfang · 2020-03-17T20:28:00Z

Technical Details

When using Azure Function and Azure SDK for Java simultaneously. It causes a dependency conflict.
For example, using Azure Function with App Configuration SDK for Java as dependencies in an independent app,
Expected behavior: running the application successfully.
Actual behavior: application crashed with unhandled exception,

[2/21/2020 9:42:14 PM] Executed 'Functions.hello' (Failed, Id=24c8735c-df65-45ea-a857-8ac83005b7fa)
[2/21/2020 9:42:14 PM] System.Private.CoreLib: Exception while executing function: Functions.hello. System.Private.CoreLib: Result: Failure
Exception: NoSuchMethodError: io.netty.handler.ssl.SslProvider.isAlpnSupported(Lio/netty/handler/ssl/SslProvider;)Z
Stack: java.lang.reflect.InvocationTargetException
[2/21/2020 9:42:14 PM]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2/21/2020 9:42:14 PM]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2/21/2020 9:42:14 PM]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2/21/2020 9:42:14 PM]  at java.lang.reflect.Method.invoke(Method.java:498)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:54)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:53)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)
[2/21/2020 9:42:14 PM]  at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)
[2/21/2020 9:42:14 PM]  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2/21/2020 9:42:14 PM]  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2/21/2020 9:42:14 PM]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2/21/2020 9:42:14 PM]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2/21/2020 9:42:14 PM]  at java.lang.Thread.run(Thread.java:748)
[2/21/2020 9:42:14 PM] Caused by: java.lang.NoSuchMethodError: io.netty.handler.ssl.SslProvider.isAlpnSupported(Lio/netty/handler/ssl/SslProvider;)Z
[2/21/2020 9:42:14 PM]  at reactor.netty.http.client.HttpClientSecure.<clinit>(HttpClientSecure.java:79)

More details of the exception can redirect to the Azure SDK for Java issue: Azure/azure-sdk-for-java#8392

Investigation

Azure Function is using GRPC-netty 1.20.0 which using Netty 4.1.34.Final. The latest version of GRPC-Netty is 1.28.0 which using version 4.1.45.Final for both artifacts, netty-codec-http2 and netty-handler-proxy in the io.netty group.

Azure SDK for Java is using Netty 4.1.44.Final. (I am from Azure SDK for Java team, we are going to upgrade to Netty 4.1.45.Final for above two artifacts, netty-codec-http2 and netty-handler-proxy in the 2020 April release, which is in the first week of April. See section Dependencies for more details on other artifacts of io.netty and version that we are going to use.).

Proposal

Upgrade GRPC-netty 1.20.0 to latest version 1.28.0 will make Azure Function and Azure SDK for Java dependency consistency because both SDKs are using the same Netty version 4.1.45 then.

Dependencies

AZURE FUNCTION dependency tree:

com.microsoft.azure.functions:azure-functions-java-worker:jar:1.5.2
[INFO] +- com.microsoft.azure.functions:azure-functions-java-library:jar:1.3.1-SNAPSHOT:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:3.7.1:compile
[INFO] +- io.grpc:grpc-protobuf:jar:1.20.0:compile
[INFO] |  +- io.grpc:grpc-core:jar:1.20.0:compile
[INFO] |  |  +- io.grpc:grpc-context:jar:1.20.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.android:annotations:jar:4.1.1.4:compile
[INFO] |  |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- io.opencensus:opencensus-api:jar:0.19.2:compile
[INFO] |  |  \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.19.2:compile
[INFO] |  +- com.google.guava:guava:jar:26.0-jre:compile
[INFO] |  |  +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] |  |  \- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
[INFO] |  \- io.grpc:grpc-protobuf-lite:jar:1.20.0:compile
[INFO] +- io.grpc:grpc-stub:jar:1.20.0:compile
[INFO] +- io.grpc:grpc-netty:jar:1.20.0:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.34.Final:compile (version selected from constraint [4.1.34.Final,4.1.34.Final])
[INFO] |  |  +- io.netty:netty-common:jar:4.1.34.Final:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.34.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.30.Final:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.30.Final:compile
[INFO] |  |  +- io.netty:netty-codec:jar:4.1.34.Final:compile
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.30.Final:compile
[INFO] |  |  \- io.netty:netty-codec-http:jar:4.1.30.Final:compile
[INFO] |  \- io.netty:netty-handler-proxy:jar:4.1.30.Final:compile
[INFO] |     \- io.netty:netty-codec-socks:jar:4.1.30.Final:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] +- commons-cli:commons-cli:jar:1.4:compile
[INFO] +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- net.java.dev.jna:jna-platform:jar:5.3.0:compile
[INFO]    \- net.java.dev.jna:jna:jar:5.3.0:compile

Azure SDK for Java dependency tree and the maven io.netty dependencies will be used in the upcoming April release.:

// Current Azure SDK's io.netty dependency version
 com.azure:azure-core-http-netty:jar:1.4.0:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.44.Final:compile
[INFO] |  |  +- io.netty:netty-common:jar:4.1.44.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.44.Final:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.44.Final:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.1.44.Final:compile
[INFO] |  |  \- io.netty:netty-codec-socks:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-buffer:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-codec-http:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-transport-native-unix-common:jar:4.1.44.Final:compile
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.44.Final:compile
[INFO] |  \- io.projectreactor.netty:reactor-netty:jar:0.9.4.RELEASE:compile

// Upcoming Azure SDK's io.netty dependency version
<dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-handler</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-handler-proxy</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-buffer</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-codec-http</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-codec-http2</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-transport-native-unix-common</artifactId>
      <version>4.1.45.Final</version> 
    </dependency>
    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-transport-native-epoll</artifactId>
      <version>4.1.45.Final</version>
      <classifier>linux-x86_64</classifier>
    </dependency>

cc: @mrm9084

The text was updated successfully, but these errors were encountered:

joshfree · 2020-03-17T20:53:55Z

The impact of this issue is that Java customers using Unified Java SDKs, like App Config,
https://azure.github.io/azure-sdk/releases/2019-11/java.html will run into the issue above.

In the short term, we should update Azure Function and Azure Core dependencies to no longer have conflicting dependencies (which the issue @mssfang filed tracks). Afterwards, we should come up with a strategy for aligning our 3rd party dependencies to avoid regressing this scenario.

mssfang · 2020-03-31T17:57:35Z

@fabiocav Can you help us with this issue?

sitase · 2020-04-02T12:39:44Z

The underlying problem is that the runtime classes of the azure functions worker pollutes classpath of the functions code. You should isolate the worker in its own classloader.

slashphi · 2020-05-01T18:28:24Z

Will there a fix for that soon? I have the same issue with gson.

fabiocav · 2020-05-01T18:39:58Z

Moved this to the java repository for proper triage.

@amamounelsayed can you please assist?

jspenc4 · 2020-05-06T21:28:02Z

isolate the worker in its own classloader? is that really the workaround?

amamounelsayed · 2020-05-06T22:26:50Z

We are working on a proposal for this issue we will test it for java11. The objective is to have the client jar take presence over java worker jars.

To upgrade java worker to the latest version of grpc we have tracking item to it, there is no ETA yet, #354

@mssfang the latest version is 1.29.0, correct? I see also there is version grpc-netty-shaded which we may explore too as this may avoid any netty issues in future.

amamounelsayed · 2020-05-06T22:31:44Z

cc @anirudhgarg

mrm9084 · 2020-05-06T22:33:45Z

Can we make sure it also works in Java 8? This was originally seen with Java 8 Functions.

amamounelsayed · 2020-05-06T22:39:22Z

@mrm9084 yes this is the plan but this the log term plan.

I believe the first step will be upgrade grpc to 1.29.0 if this work with SDK Java instead of 1.28.0.
Second we will explore the grpc-netty-shaded version so by this way there is no netty conflits.
Third work on the classloader proposal and test with java11.
Fourth add it to java8.

Thank you so much

JarroVGIT · 2020-05-07T14:35:58Z

Hi @amamounelsayed , thanks for the update. I don't want to push anything but this is currently a major problem in our situation as we cannot interact with any other Azure services (authenticated) in Azure Functions. My question would be; when can we expect this to be released (roughly)? This issue is here since February (looking at this issue), and it currently stops us from using Azure Functions. That seems dramatic maybe, but we are only able to use Python or Java and Python is not supported in Windows and only Windows has Vnet-integration (which is a must-have for us).

Thanks in advance!

mssfang · 2020-05-07T14:49:08Z

@amamounelsayed Yes. They released a new version. So the latest is 1.29.0

jspenc4 · 2020-05-07T16:55:29Z

Just wanted to let you know, turns out my problem is actually a conflict between the netty version required by aws sdk and azure sdk. So i was able to solve that by excluding the aws sdk netty dependencies. Jim

…

On Wed, May 6, 2020 at 3:39 PM amamounelsayed ***@***.***> wrote: @mrm9084 <https://github.com/mrm9084> yes this is the plan but this the log term plan. I believe the first step will be upgrade grpc to 1.29.0 if this work with SDK Java instead of 1.28.0. Second we will explore the shaded version so by this way there is no netty conflits. Third work on the classloader proposal and test with java11. Fourth add it to java8. Thank you so much — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#365 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFC4EE63CQBDNWFB26WGLALRQHRKPANCNFSM4MXJBC4Q> .

amamounelsayed · 2020-05-08T18:59:12Z

@JarroVGIT Thank you for your feed back, we will get back to you with ETA as soon as we can.

liuhegui · 2020-05-11T00:52:00Z

When I tried to use WebClient to get oauth token, I got same issue.

Here is my issue webclient

joshfree · 2020-05-20T23:57:16Z

@alzimmermsft can you share more details on the Azure SDK for Java Parent BOM file that Azure Functions Java Worker should take a dependency on to unify on standard 3rd party dependencies?

TsuyoshiUshio · 2020-05-21T00:19:53Z

Hi @mssfang We are working on this issue. Probably we are going to shaded package of the netty.
I'd like to provide a workaround to the customer until it is released. Could you tell me which azure-core-http-netty has the same version as the current Azure Functions Java Worker?

mssfang · 2020-05-21T05:19:59Z

@TsuyoshiUshio I don't think we have a version of azure-core-http-netty is using GRPC-netty 1.20.0. But @alzimmermsft can confirm that.

alzimmermsft · 2020-05-21T16:28:55Z

azure-core-http-netty utilizes the Netty libraries directly, here is a quick version matching output.

`azure-core-http-netty` version	`grpc-netty` version`	Shared Netty version
1.5.0	1.28.0 or 1.28.1	4.1.45
1.0.0 - 1.3.0	1.25.0 - 1.27.2	4.1.42

I would recommend using the azure-core-http-netty 1.5.0 and either grpc-netty 1.28.0 or 1.28.1 as we've recently shipped a dependency management BOM, azure-sdk-bom (1.0.0 (some work still in progress), which declared said azure-core-http-netty and the Azure SDKs libraries which shared dependency version compatibility.

amamounelsayed · 2020-05-21T16:57:12Z

Thank you @alzimmermsft @mssfang @joshfree

One of the proposed solution we are exploring is to use grpc-netty-shaded
Here is the dependency tree if azure functions worker uses netty shaded.
I belive this can solve the netty issue dependency and shouldn't be conflicts in future.
@alzimmermsft @mssfang Do you agree? Also is there any other jars conflicts example do you use grpc-netty-shaded, commons-lang3.jar or any other dependency?

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ azure-functions-java-worker ---
[INFO] com.microsoft.azure.functions:azure-functions-java-worker:jar:1.5.3
[INFO] +- com.microsoft.azure.functions:azure-functions-java-library:jar:1.3.1:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:3.7.1:compile
[INFO] +- io.grpc:grpc-protobuf:jar:1.20.0:compile
[INFO] | +- io.grpc:grpc-core:jar:1.20.0:compile
[INFO] | | +- io.grpc:grpc-context:jar:1.20.0:compile
[INFO] | | +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] | | +- com.google.android:annotations:jar:4.1.1.4:compile
[INFO] | | +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] | | +- io.opencensus:opencensus-api:jar:0.19.2:compile
[INFO] | | - io.opencensus:opencensus-contrib-grpc-metrics:jar:0.19.2:compile
[INFO] | +- com.google.guava:guava:jar:26.0-jre:compile
[INFO] | | +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] | | - com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
[INFO] | - io.grpc:grpc-protobuf-lite:jar:1.20.0:compile
[INFO] +- io.grpc:grpc-stub:jar:1.20.0:compile
[INFO] +- io.grpc:grpc-netty-shaded:jar:1.20.0:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] +- commons-cli:commons-cli:jar:1.4:compile
[INFO] +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] | - org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- net.java.dev.jna:jna-platform:jar:5.3.0:compile
[INFO] | - net.java.dev.jna:jna:jar:5.3.0:compile
[INFO] - org.jmockit:jmockit:jar:1.49:test
[INFO] - com.google.code.findbugs:jsr305:jar:3.0.2:compile

alzimmermsft · 2020-05-21T17:31:32Z

@amamounelsayed the Azure SDKs under the com.azure group don't share any first level dependencies listed in that tree. This pages gives a quick overview on the primary dependencies we are allowed to use, https://azure.github.io/azure-sdk/java_implementation.html#java-dependencies-approved-list.

The common dependencies that we will used are the following:

Reactor
Reactor Netty
Netty
OkHttp
Jackson
SLF4J
OpenTelemetry
JUnit 5 (Test only)

Netty and OkHttp depend on which HttpClient implementation package is being used, or neither is a custom implementation is being used.

anuchandy · 2020-05-27T19:44:37Z

@amamounelsayed shading Grpc + it's dependencies linked to the function java worker seems a good option. I guess if we choose this path then you'll also need to "relocate" those conflicting dependencies (e.g. netty). If it helps, this comment has a downloadable sample showing how to do the relocation of dependencies - Azure/azure-sdk-for-java#11104 (comment)

mssfang changed the title ~~[Dependency Conflict] Azure Function has a dependency conflict with Azure Core~~ [Dependency Conflict]Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java Mar 17, 2020

mssfang changed the title ~~[Dependency Conflict]Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java~~ Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java Mar 17, 2020

fabiocav transferred this issue from Azure/Azure-Functions May 1, 2020

ghost added the Needs: Triage (Functions) label May 1, 2020

JarroVGIT mentioned this issue May 6, 2020

[BUG] Using azure.storage.blob together with azure.identity in Azure Functions results in NoSuchMethodError Azure/azure-sdk-for-java#10666

Closed

3 tasks

amamounelsayed self-assigned this May 6, 2020

amamounelsayed added triaged and removed Needs: Triage (Functions) labels May 6, 2020

anuchandy mentioned this issue May 8, 2020

[BUG] java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop. calling azure blob storage Azure/azure-sdk-for-java#10948

Closed

3 tasks

amamounelsayed assigned TsuyoshiUshio May 12, 2020

TsuyoshiUshio mentioned this issue May 13, 2020

Upgrade GRPC to 1.29.0 #370

Merged

anuchandy mentioned this issue May 26, 2020

Azure-Functions broke Storage API Azure/azure-functions-java-library#110

Closed

amamounelsayed mentioned this issue May 27, 2020

Epoll based Netty Transport error in Function Development Tool on Linux #373

Closed

amamounelsayed mentioned this issue May 29, 2020

Add shaded netty #374

Merged

amamounelsayed mentioned this issue Jun 19, 2020

Feature/shadding #379

Merged

amamounelsayed closed this as completed in #379 Jun 24, 2020

mssfang mentioned this issue Jun 30, 2020

[BUG] App Configuration doesn't work in Azure Function Azure/azure-sdk-for-java#8392

Closed

3 tasks

amamounelsayed mentioned this issue Jul 2, 2020

Azure Functions Java Worker 1.7.1 -- Dependencies change opt in #381

Closed

EthanNguyenDev mentioned this issue Oct 24, 2020

Azure Function with Spring Cloud App Config - Netty conflict Azure/AppConfiguration#424

Closed

vdiddlodka mentioned this issue Aug 24, 2021

Netty conflict with azure-storage-blob Azure/azure-functions-java-library#155

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java #365

Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java #365

mssfang commented Mar 17, 2020

joshfree commented Mar 17, 2020

mssfang commented Mar 31, 2020

sitase commented Apr 2, 2020

slashphi commented May 1, 2020

fabiocav commented May 1, 2020

jspenc4 commented May 6, 2020

amamounelsayed commented May 6, 2020 •

edited

Loading

amamounelsayed commented May 6, 2020

mrm9084 commented May 6, 2020

amamounelsayed commented May 6, 2020 •

edited

Loading

JarroVGIT commented May 7, 2020

mssfang commented May 7, 2020

jspenc4 commented May 7, 2020 via email

amamounelsayed commented May 8, 2020

liuhegui commented May 11, 2020 •

edited

Loading

joshfree commented May 20, 2020

TsuyoshiUshio commented May 21, 2020 •

edited

Loading

mssfang commented May 21, 2020

alzimmermsft commented May 21, 2020 •

edited

Loading

amamounelsayed commented May 21, 2020 •

edited

Loading

alzimmermsft commented May 21, 2020

anuchandy commented May 27, 2020

Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java #365

Java 3rd Party dependency conflict between Azure Function and Azure SDK for Java #365

Comments

mssfang commented Mar 17, 2020

Technical Details

Investigation

Proposal

Dependencies

joshfree commented Mar 17, 2020

mssfang commented Mar 31, 2020

sitase commented Apr 2, 2020

slashphi commented May 1, 2020

fabiocav commented May 1, 2020

jspenc4 commented May 6, 2020

amamounelsayed commented May 6, 2020 • edited Loading

amamounelsayed commented May 6, 2020

mrm9084 commented May 6, 2020

amamounelsayed commented May 6, 2020 • edited Loading

JarroVGIT commented May 7, 2020

mssfang commented May 7, 2020

jspenc4 commented May 7, 2020 via email

amamounelsayed commented May 8, 2020

liuhegui commented May 11, 2020 • edited Loading

joshfree commented May 20, 2020

TsuyoshiUshio commented May 21, 2020 • edited Loading

mssfang commented May 21, 2020

alzimmermsft commented May 21, 2020 • edited Loading

amamounelsayed commented May 21, 2020 • edited Loading

alzimmermsft commented May 21, 2020

anuchandy commented May 27, 2020

amamounelsayed commented May 6, 2020 •

edited

Loading

amamounelsayed commented May 6, 2020 •

edited

Loading

liuhegui commented May 11, 2020 •

edited

Loading

TsuyoshiUshio commented May 21, 2020 •

edited

Loading

alzimmermsft commented May 21, 2020 •

edited

Loading

amamounelsayed commented May 21, 2020 •

edited

Loading