Do we need real device tests?

[philipphofmann]

This discussion tries to answer the following question: Do we need to run our UI tests on real devices, or are simulators enough?

Background

With the release of APM in 7.2.0, we had plenty of hotfixes afterward. One of the many mitigations to avoid such issues and other regressions in the future was to add UI tests and run them on SauceLabs.

Flaky Test

We have the problem that the UI tests fail around 20% of the time. The main reason is that the testNavigationTransaction fails with

Failed to get matching snapshots: Lost connection to the application (pid 16212). (Underlying Error: Couldn’t communicate with a helper application. Try your operation again. If that fails, quit and relaunch the application and try again. The connection to service created from an endpoint was invalidated: failed to check-in, peer may have been unloaded: mach_error=10000003.)

The test fails randomly no matter which OS or iPhone. I ran the UI tests repeatedly 100 times on an iPhone 12 with iOS 15.2 and an iPhone 6 with iOS 12.5.5 and the testNavigationTransaction didn't fail a single time.

Past Issue Analysis

This section looks at past issues and identifies which one of these could have been caught by UI tests on a simulator and which ones by UI tests on a real device.

Issue	Simulator	Real Device
fix: Callback issue for auto performance (#1275)	yes	yes
fix: Crash when swizzling Nib UIViewController (#1277)	yes	yes
fix: Swizzling crash on iOS 13 (#1297)	yes	yes
fix: Crash for Custom ViewController init on iOS 15 (#1361)	yes	yes
fix: Crash for Call Should be on Main Thread (#1371)	yes	yes
fix: Remove Check for Original Method Call When Swizzling (#1383)	yes	yes
fix: Crash when Getting Subclasses (#1396)	yes	yes
fix: Crash in Span when Tracer nil (#1416)	yes	yes
fix: Crash when objc_getClassList returns different values (#1420)	yes	yes

As it turns our all issues that we could have caught with UI tests can be detected on a simulator and don't need a real device. We only need to make sure we run the UI tests on different iOS versions.

[philipphofmann]

We can't test iOS 11 and 10 on simulators because as of 21st December 2021 we can't run macos-10.14 on GitHub Actions and macos-10.15 only supports simulators with iOS 12 and higher. As we also want to test iOS 11 and 10 we need real device tests.

[brustolin]

Although Apple recommends testing apps in real device before publishing, in 11 years of iOS development I never found any difference running an app in a real device or simulator. I never caught a bug in a real device that didn't occur in simulation.

[armcknight]

I've maybe had a handful of cases where it made sense to test with a real device, like for stack unwinding edge cases that needed to be validated on an ARM chip vs x86_64, or something to do with AVFoundation/location that wasn't able to be done in a simulator at the time–they're always gaining more simulation capability.

Maybe nowadays with an M1 desktop running simulators in CI that's no longer an issue. Anyways, now that we have real device tests, it's something we can more easily deploy to reproduce/validate a bug/fix if we need it. I think we can close this discussion.

[philipphofmann]

Yes, I think we can close it.