RecommendationService crashes under load-testing - "Request had insufficient authentication scopes."

[Boenan]

I am currently using this application to evaluate communication performance between the nodes in Kind. I used the Locust test file located in LoadGenerator but changed to a constant delay of 3 seconds between executing new task. The load-testing was set up with a total of 2000 simulated users with an incremental rate of 2 users per second. When reaching about 1000 simulated users the RecommendationService pod begins to crash. I suspect that when the RecommendationService receives too many request it crashes.

Inspecting the log displays the following error messages:

HttpError: <HttpError 403 when requesting https://clouddebugger.googleapis.com/v2/controller/debuggees/register?alt=json returned "Request had insufficient authentication scopes.">

WARNING:googleapiclient.http:Encountered 403 Forbidden with reason "insufficientPermissions"
I0608 13:13:58.284570    11 gcp_hub_client.py:333] Failed to register debuggee: {'debuggee': {'project': u'jovial-lamp-270413', 'labels': {'projectid': u'jovial-lamp-270413', 'version': '1.0.0', 'module': 'recommendationserver'}, 'uniquifier': 'ad449bd3fa6bdb48691966e018561b663ea695dc', 'description': u'jovial-lamp-270413-recommendationserver-1.0.0', 'agentVersion': 'google.com/python27-gcp/v2'}}, Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/googleclouddebugger/gcp_hub_client.py", line 317, in _RegisterDebuggee
    response = service.debuggees().register(body=request).execute()
  File "/usr/local/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 851, in execute
    raise HttpError(resp, content, uri=self.uri)

I'm not sure what to do to solve this problem. I have tried deploying multiple RecommendationService and ProductCatalogService replicas to circumvent this problem, but with no result.

[askmeegs]

Hi Fabian, I'm not sure that this is a loadgen issue, but rather a GCP auth issue, but not sure.

Because you're running in KIND, and Google Cloud debugger is enabled for the app, you're seeing auth errors because your pods are trying to connect to GCP with no auth. You need a service account token for GCP inside your pod. In GKE, this happens by default, but in non-GCP K8s environments, you have to do that manually. So you'll likely have to do one of two things -

1. Turn off Google Cloud debugger for the app by uncommenting this line in your deployment YAML
2. Create a service account in GCP, give it debugger IAM permissions, and mount a service account key into your pods. more info here.

TODO on our end - improve instructions for GCP service auth when running this app outside GKE.

[askmeegs]

Closing this in favor of the larger tracking bug for kustomize support, including for off GCP development (eg. Kind). #536