Session Aggregates are randomly dropped

[Dav1dde]

Session aggregates are being dropped/lost before the reach Kafka/Storage.

We could verify by peeking into the Kafka Topic that the aggregates (as metrics) never make it into Kafka.

From the Kafka Topic:

 (datetime.datetime(2024, 1, 18, 10, 9), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 11), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 12), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 14), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 17), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 19), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 21), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 23), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 25), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 29), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 31), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 32), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 34), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 35), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 40), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 49), 30.0),
 (datetime.datetime(2024, 1, 18, 10, 56), 30.0)]

In the UI:

Expected to see one entry every minute with a value of 30.

Script which generates an envelop containing a session aggregate every minute:

#!/usr/bin/env bash


URL="https://<INGEST URL>/api/<PROJECT ID>/envelope/?sentry_key=<KEY>&sentry_version=7&sentry_client=sentry.javascript.browser%2F7.93.0"

RELEASE="r1"
ENVIRONMENT="production"
USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

while true; do
    DT=$(date -u +'%Y-%m-%dT%H:%M:%S.398Z')
    echo $DT

    PAYLOAD="{\"sent_at\":\"${DT}\",\"sdk\":{\"name\":\"sentry.javascript.browser\",\"version\":\"7.93.0\"}}
{\"type\":\"sessions\"}
{\"attrs\":{\"release\":\"$RELEASE\", \"environment\":\"$ENVIRONMENT\", \"user_agent\":\"$USER_AGENT\"},\"aggregates\":[{\"started\":\"$(date -u +'%Y-%m-%dT%H:%M:00.000Z')\",\"exited\":100}]}"

    echo $PAYLOAD 
    curl -H 'User-Agent:' -H 'Accept:' -H "Content-Type: application/x-sentry-envelope" -X POST "$URL" -d "$PAYLOAD"

    echo ""
    sleep 60
done

I was not able to reproduce this with a local Sentry and Relay instance, Relay consistently delivered the metrics into the Kafka topic.

The local instance had the following non-standard settings:

SENTRY_FEATURES['organizations:metrics-extraction'] = True
SENTRY_FEATURES['organizations:transaction-metrics-extraction'] = True
SENTRY_FEATURES['organizations:custom-metrics'] = True
SENTRY_FEATURES['organizations:ddm-ui'] = True
SENTRY_FEATURES['organizations:release-health-drop-sessions'] = True

SENTRY_RELEASE_HEALTH = "sentry.release_health.metrics.MetricsReleaseHealthBackend"
SENTRY_RELEASE_MONITOR = "sentry.release_health.release_monitor.metrics.MetricReleaseMonitorBackend"

I manually verified that the sessionMetrics setting in the project config matches the production one version: 2, drop: true.

[Dav1dde]

Ingesting the session aggregates directly into processing relays (skipping pops) with the same above script does not show this issue (using a value of 100 instead of 30):

The test was performed by sending the payload to a single Relay instance everytime, this does not rule out that there may be a problem with non-cached Project Configs.

[Dav1dde]

Still no issues when running it against random processing relays (should rule out issues with non-cached project configs):

[Dav1dde]

It appears like running the script against a single pop instance also works.

[Dav1dde]

Running the script against random pop instances shows the same behaviour:

[jjbayer]

Did not notice anything obvious in the code (session aggregate -> aggregation -> submission via global metrics endpoint), will try to repro the behavior with an integration test now.

[jjbayer]

There's definitely an edge case in which we aggregate & attempt to send metrics even though the project has expired (see #2987), but I doubt that it explains the behavior above.

[jjbayer]

Seems fixed by #2987:

Problem seems to have been project configs not being actively fetched when the only incoming traffic (from processing relay's point of view) is metrics via the global endpoint. See linked PR for details.

[Dav1dde]

Waiting for customer to confirm that the issue is fixed for them.