Mobile SDK overhead measurement

[bruno-garcia]

What we want to measure:

• How the SDK affects Binary size
• How the SDK affects the app startup time
• Common SDK "features", like captureMessage,captureException, startTransaction, addBreadcrumb, performance profiling
• How the SDK affects the app memory footprint?

What do we measure against:

• The baseline is an app without sentry
• We measure the overhead on the main branch and store it somewhere (gcloud)
• We measure the overhead on a branch which has an open PR against the main branch

---

• Work with the profiling team since they are working on this in the context of profiling
• Bring the items from notion into GH
• Define scope
• measure screenshot overhead
• Compare the result with main and post in the PR as a comment. See action

General approach

The goal is to run benchmarking on Sauce labs and check what’s the overhead (e.g. of the SDK during App start).
We can break the build if the overhead is more than a magic number or a specific percentage.
Profiling benchmark in Java does that - they break the test if the overhead is more than 5%. They also break the build if the overhead exceeds an absolute threshold.

Related:

• Example comment with the snapshot comparison similar to what the JS SDK does. Example JS that posts to PR feat(angular): Add Angular 14 support sentry-javascript#5253 (comment)
• This repo has profiling for MAUI vs Xamarin and is based on a script on adb running the app and measuring startup based on logcat printed by the app itself. Could be used as a reference: https://github.com/jonathanpeppers/maui-profiling
• https://github.com/getsentry/sentry-javascript/blob/master/.github/workflows/build.yml#L174-L206
• feat(angular): Add Angular 14 support sentry-javascript#5253 (comment)
• https://getsentry.atlassian.net/browse/MOBILE-253
• https://netflixtechblog.com/fixing-performance-regressions-before-they-happen-eab2602b86fe

---

Tracking issues, PRs and the current status

Java

• App size: [Android]: Report SDK size as a comment in PRs sentry-java#1303
• Startup time: Measure Sentry.init impact on app startup time sentry-java#1965
• Features: Compare benchmarks with previous versions sentry-java#2037

The recently added (profiling-related) in-app benchmarks are based on Espresso (see getsentry/sentry-java#2013 & getsentry/sentry-java#2077. There has been a discussion about using Macrobenchmark in the PR that introduced the benchmarks, with the result being inconclusive and going ahead with the in-app benchmarks for now but evaluate Macrobenchmark.

Additionally, there's been an experiment with Jetpack Macrobenchmark in GDP to measure app startup - getsentry/sentry-android-gradle-plugin#317

[marandaneto]

@vaind would you mind updating this issue with the latest changes?
Some checkboxes are done already, and some issues can be closed I suspect.

[marandaneto]

The size analyses bit could be replaced by https://docs.emergetools.com/reference/get_analysis
Not telling that we should do this, just sharing that some tools are getting specialized in this field.