Unable to create credentials for gcp bucket

[andriihomiak]

STR

1. Create a bucket with gcp provider

$ neuro blob mkbucket                    
 Id            bucket-43071007-89bc-452f-a1d5-176fe9260778                              
 Uri           blob://default/andriikhomiak/bucket-43071007-89bc-452f-a1d5-176fe9260778 
 Org name      NO_ORG                                                                   
 Project name  andriikhomiak                                                            
 Created at    a moment ago                                                             
 Provider      gcp                                                                      
 Imported      False                                                                    
 Public        False 

2. Try creating credentials for this bucket and observe the error

$ neuro blob mkcredentials bucket-43071007-89bc-452f-a1d5-176fe9260778 
ERROR: Illegal argument(s) (Unexpected exception: <HttpError 403 when requesting 
https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak93fe06@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Permission 'iam.serviceAccountKeys.create' denied onresource (or it may not exist).". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 
'IAM_PERMISSION_DENIED', 'domain': 'iam.googleapis.com', 'metadata': {'permission': 
'iam.serviceAccountKeys.create'}}]">. Path with query: /api/v1/buckets/persistent_credentials.)

Buckets API logs provide the following:

2023-06-14 16:23:20,554 - googleapiclient.http - WARNING - Encountered 403 Forbidden with reason "PERMISSION_DENIED"
2023-06-14 16:23:20,554 - root - ERROR - Unexpected exception: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak0f33a8@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Permission 'iam.serviceAccountKeys.create' denied on resource (or it may not exist).". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'IAM_PERMISSION_DENIED', 'domain': 'iam.googleapis.com', 'metadata': {'permission': 'iam.serviceAccountKeys.create'}}]">. Path with query: /api/v1/buckets/persistent_credentials.
Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 853, in handle_exceptions
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 760, in create_credentials
    credentials = await self.credentials_service.create_credentials(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/service.py", line 256, in create_credentials
    role = await self._provider.create_role(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 1008, in create_role
    resp = await self._create_sa_key(full_name=self._make_sa_full_name(username))
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 776, in _wrapper
    return await loop.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 923, in _create_sa_key
    self._iam_client.projects()
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak0f33a8@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Permission 'iam.serviceAccountKeys.create' denied on resource (or it may not exist).". Details: "[{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'IAM_PERMISSION_DENIED', 'domain': 'iam.googleapis.com', 'metadata': {'permission': 'iam.serviceAccountKeys.create'}}]">

Occasionally the following also happens instead of the 403 (unable to reliably reproduce):

$ n blob mkcredentials bucket-ebd31c57-1c19-42d2-a67d-1e3ed0c747dc
ERROR: Illegal argument(s) (Unexpected exception: <HttpError 404 when requesting 
https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak3552a0@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Unknown service account". Details: "Unknown service account">. Path with query: /api/v1/buckets/persistent_credentials.)

The logs then look this way:

2023-06-14 16:18:59,231 - root - ERROR - Unexpected exception: <HttpError 404 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak3552a0@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Unknown service account". Details: "Unknown service account">. Path with query: /api/v1/buckets/persistent_credentials.
Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 853, in handle_exceptions
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 760, in create_credentials
    credentials = await self.credentials_service.create_credentials(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/service.py", line 256, in create_credentials
    role = await self._provider.create_role(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 1008, in create_role
    resp = await self._create_sa_key(full_name=self._make_sa_full_name(username))
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 776, in _wrapper
    return await loop.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 923, in _create_sa_key
    self._iam_client.projects()
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak3552a0@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Unknown service account". Details: "Unknown service account">

In both cases, the service accounts are created and are visible in the GCP dashboard

[andriihomiak]

Upon further inspection it is clear that this is due to missing iam.serviceAccountKeys.create permission

[andriihomiak]

After the issue with the SA was resolved now occasionally the request fails with 404:

$ neuro blob mkcredentials   bucket-ebd31c57-1c19-42d2-a67d-1e3ed0c747dc
 Id          bucket-credentials-d86346e0-503a-4c4b-83b1-67021fab3a40 
 Read-only:  False                                                   
Credentials for bucket 'bucket-ebd31c57-1c19-42d2-a67d-1e3ed0c747dc':
bucket_name: neuro-pl-<...>
key_data: <...>
project: <...>

$ neuro blob mkcredentials   bucket-ebd31c57-1c19-42d2-a67d-1e3ed0c747dc
ERROR: Illegal argument(s) (Unexpected exception: <HttpError 404 when requesting 
https://iam.googleapis.com/v1/projects/.../serviceAccounts/bkt-user-...@developmaccount">. Path with query: /api/v1/buckets/persistent_credentials.)

Logs of the API (both replicas) show the same error as described above:

2023-06-14 16:18:59,231 - root - ERROR - Unexpected exception: <HttpError 404 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak3552a0@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Unknown service account". Details: "Unknown service account">. Path with query: /api/v1/buckets/persistent_credentials.
Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 853, in handle_exceptions
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/api.py", line 760, in create_credentials
    credentials = await self.credentials_service.create_credentials(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/service.py", line 256, in create_credentials
    role = await self._provider.create_role(
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 1008, in create_role
    resp = await self._create_sa_key(full_name=self._make_sa_full_name(username))
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 776, in _wrapper
    return await loop.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.local/lib/python3.9/site-packages/platform_buckets_api/providers.py", line 923, in _create_sa_key
    self._iam_client.projects()
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://iam.googleapis.com/v1/projects/REDACTED/serviceAccounts/bkt-user-andriikhomiak3552a0@REDACTED.iam.gserviceaccount.com/keys?alt=json returned "Unknown service account". Details: "Unknown service account">