App/Department teams using terraform within Project factory Projects. #1830
Replies: 4 comments 8 replies
-
|
Jack, thanks a lot for starting this discussion and apologies for being a bit late. This is something which has been on our radar for a while, and which we did not fully flesh out yet. One approach we were considering is having team-level IaC projects, so as to detach from the org-level one and be able to create application-level service accounts outside of the scopes controlled by those same accounts. Which is similar to what you suggest, except for where those service accounts and buckets are created. Let's try and discuss this here so we get to a design that satisfies all requirements. @drebes who has ideas on this, @sruffilli and @juliocc (currently on vacation but back soon) |
Beta Was this translation helpful? Give feedback.
-
|
PR #2505 adds a set of sample files and supporting documentation that explains how to implement teams via the project factory. Some of the low level substitutions for ids are implemented in the PR, but the general approach is already usable and described in the new project factory README. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for starting this thread. I think we can record here some of the internal ideas we have been discussing so far and share with a broader audience. In the end, I think this is a general discussion on what level of abstraction should your platform engineering team provide as a service to the workload teams to build upon. The way I see there's generally 3 (but two reallistically for this discussion if the focus is more on core infra versus apps):
I don't think there's "right" or "wrong", as it depends a lot on industry and homogeneity expected from the stack. I personally worked a lot with customers trying to do 1 and 3, while FAST defaults comes mostly for customers doing 2. We did discuss extending PF to follow more of 1, though, and the general idea would be to initially prototype the concept in I generally see customers preferring 2 versus 1 (including in some very regulated industries) due to the fear/concern that centralizing project creation via centrally governed factories that do all would slow down teams. I understand the feeling if you're coming from some legacy where centralized means slow, but central process does not mean central executuion, and this is only a risk if you don't have a mature concept for the project factory that supports all that the workload teams may need (as the platform evolves). |
Beta Was this translation helpful? Give feedback.
-
|
Does anyone know of an example of terraform code of the 1st option here? |
Beta Was this translation helpful? Give feedback.
-
|
@jaclyntaylor7 since i asked this original question, a lot of work has been done on project factory: The module now has some expanded support for defining TF backends and Service accounts: https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/v33.0.0/modules/project-factory#automation-project-and-resources I've not personally had a chance to try out this latest addition, but am looking forward to giving it a shot soon. Best of luck. |
Beta Was this translation helpful? Give feedback.
-
|
I have been trying this out as per link you shared. First thing I encountered was just the order of resource creation. I needed to make sure I created the "controlling project" BEFORE applying it to another project as an automation . Without this order of operations, the TF apply on stage 2-project-factory would fail if the controlling project didn't exist. Perhaps there is a clever way to specify a dependency but I haven't seen a simply way to do that yet. |
Beta Was this translation helpful? Give feedback.
-
|
correct, there's no simple way to do it and it's a simple enough process anyway :) |
Beta Was this translation helpful? Give feedback.
-
|
Looks like the imple way would be to use multiple project factories. The first would create the IaC project and the latter would use them |
Beta Was this translation helpful? Give feedback.
-
|
I've actually encountered basically all of the above with customers. Some of our customers are fully GKE everything, so per app team projects are mostly non existent besides 1-2 who need cloud SQL or similar which they self manage, including their own TF backends, which was some extra TF the customer made themselves. Others have not fully adopted TF in their app teams, and so simply provide projects via the normal project factory, where teams can elect to clickops, or terraform as suits them best. Another, wrote a bunch of scripts to generate backends, providers, and the PF pipeline will trigger a stage to create them a git repo for terraform pre populated with their provider and SA etc and a matching pipeline file. That was custom built again by the customer so not something i can commit back. Catering to all is definitely unreasonable. I follow your thinking Roberto, create a PF for prod/nonprod, and when i create the projet i add an SA and a bucket. It works fine most of the time with very little code extension needed. It also means the application TF stack doesn't have to include their own bootstrapped backends, so they can run TF destroys if they need to, without having to mess with the backends. Nice to hear the thoughts here, though its hard to pin down a single strategy. |
Beta Was this translation helpful? Give feedback.
-
|
Last question that came to mind perhaps for @drebes what are your thoughts around the workload identity pool on an app level. Do you allow them to hook into the fabric one, or encourage app teams to create their own pool for their own repos? I think this perhaps feeds into your 1 vs 2 workflows. Perhaps in 1, everything is central,including WIF access, and in 2, that gets distributed down at a per team level. This is giving me great insights into where i need to embellish on top of fabric for each customer, so i really appreciate the discussion. Obviously it's friday night so i'm not expecting anything for now :) |
Beta Was this translation helpful? Give feedback.
-
|
Exactly. The question regarding workload identity pools in the end are analogous to how much you (or your customer) accepts to delegate to individual teams. If you want the very standardized (governed/"secure") approach with a project factory (approach 1 above) you should also manage identity pools for CI/CD integration. If you accept more team control (approach 2), you could delegate it. But one important thing to remember and frequently overlooked is that delegating individual teams the permission to manage workload identity pools directly is analogous to allowing them to use any identity in IAM policies (that is, disabling domain-restricted sharing). The reason is that workload identity pool identities are considered part of the domain, but ultimately the actual identities used are controlled on the IdP outside of Cloud Identity, which whoever manages the pool can control). |
Beta Was this translation helpful? Give feedback.
-
|
Forgot to mention: if you want to control workload identity pool only through standardized process, you should also look into the |
Beta Was this translation helpful? Give feedback.
-
Hey,
I'm curious to know what the intent is around a fully deployed fabric environment, and how individual application teams, who have a teams folder space created, and perhaps several projects via the project factory are supposed to utilize terraform in their own workspaces.
The resman layer creates a prod service account and storage bucket with very high level roles such as owner and project manager at the named team level. This controls both dev and prod however, and we can delegate local terraform running to this via the impersonation group.
Is this intended to be the terraform foundational back end for those teams? Using the provider output into iac-core outputs folder?
Or is the intent that during a project factory definition, a team creates its own terraform service accounts unique and scoped per project, and creates its own storage backends? I know this would cause some issues around terraform destroys if their own backends were included.
A few customers have remarked about a dislike of a single prod account/backend for both dev/prod folders within a named team.
It would be good to have some insight from others within partners/google who have used this in anger and at scale what the solutions were, perhaps im missing something simple.
My initial thinking is, as a platform team/provider, id want to at minimum maintain control of state backends & top level service accounts, but enable development teams to utilize a provider for their infra stack within a project. But a dev/prod split between those SA's and backends would be critical.
Unless im missing a detail, perhaps a project factory entry could have some kind of terraform bool, and if true, creates a project scoped SA & Bucket within the project, and a provider entry for the corresponding team to consume?
Sorry if i've missed a detail and looking forward to your answers and insights!
Also, a big thanks for opening up this discussion forum, I think it could be of huge value, particularly for partners, but also customers to share and discuss implementation details, future expansions, and how best to utilise these tools! A key thing that i feel was missing and could be immensely valuable to smoothing the adoption of GCP for hundreds of customers! Thanks!
Beta Was this translation helpful? Give feedback.
All reactions