-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
[WIP] Add count on module #13855
[WIP] Add count on module #13855
Conversation
Thanks for taking this on, @Pryz! I expect this will be some pretty sprawling work when done, touching lots of aspects of Terraform's behavior, so I just want to be explicit that we might need to take a few iterations here to make sure the design fits in well with other Terraform features. We'd love to work with you on this though, if you're willing. Please let us know if you want some pre-review on any intermediate steps here, or if you have any design ideas you'd like a second opinion on; we want to be respectful of your time and effort while working on this. |
Thanks @apparentlymart ! I'm totally aware that it will take some iterations to make it right :) This feature will be a big improvement for my team. Since we are using the same code base for all our environment (only variables differ) having count on module will bring more flexibility. I'm going to add some tests and will ask for review :) |
Before moving forward and start working on the graph generation I have some questions :
|
Hi @Pryz! Good questions. 😀 For Doing it at the configuration layer will have a few limitations:
The idea of starting off with support for only count as zero or one is an interesting thought. However, based on the above I think it would end up being a bit of a "dead end" that would require most of the work to be thrown away and redone in order to support higher counts, and would be challenging to use effectively since configuration would never be able to reference outputs of any module that might be If you're interested in digging in to learn more about how I hope this is helpful! Please let me know if you'd like me to elaborate on anything here. |
Thanks for implementing this. As a note, this will also allow recursion in modules, as the count will allow for the base case to be defined and interpolation will allow for the difference in passing the parameters. I would really love to have this as a side-effect as it would allow to create modules that have lists and maps as input, and can create nested objects from them. An example would be aws_api_gateway_resource where it's easy to see them being nested. But checks have to be put in place to prevent infinite recursions that might cost a lot (or a huge runtime till terraform crash). A maximum module depth would do the trick for this. |
Thanks @apparentlymart for all those details. Going to start working on the next iteration now :) |
Hi again @Pryz! I happened to be in the graph-building code today for a different reason and realized that my last comment contained incorrect information. I'd noticed that modules are not actually subgraphs in the sense that I thought I remembered them being, but are instead just all flattened into a single graph by Unfortunately this means that even after the first step of graph construction we've already "lost" the module boundaries, with the module structure then just implied by the Unfortunately I don't have a good immediate suggestion for how to proceed here. Terraform's current architecture is making a bunch of assumptions that make count on modules rather hard to implement. If you'd like to see more details on how the graph is constructed I suggest starting at |
The modification of the Path needs to happen before : https://github.com/hashicorp/terraform/blob/master/terraform/transform_config.go#L101. One solution I see here is to deal with I feel like doing so will simplify also things like Outputs since when we create the context, the module.Tree will be kind of definitive. Also Something I don't get is why |
Consider the requirement that For resources, this is possible because they exist in configuration (so we can prove that the interpolation variable is valid). If we handle If we instead find a way to deal with it during graph construction then the module will still be present in config but its graph nodes elided. This way the interpolation system can still see that the block exists in config. We would need to take some special care during graph construction to make sure the dependencies get wired together properly for the input variables and outputs; resources are represented by a single graph node for the entire set of instances (using Note that modules do not use subgraphs because subgraphs are, from the perspective of the main graph, just single nodes that must complete in their entirety before their dependencies can run. Since modules can be large and have many resources, we want them flattened into a single top-level graph to get the parallelism that results when e.g. a resource depends on only one of several outputs in a module. One way we could get around this is to do it in config as you suggested and then add some extra bookkeeping info to |
Ok, I need to fix the failing test but here is basically the prototype. The idea is to take care of the count during the load of the configuration. We end up with something like (and I will add tests to prove it), for example with a
Going to try to add more feature tests. @apparentlymart any thought ? :) |
This seems like a good start, @Pryz! I think next it would be good to figure out how interpolation for
I think the last of these will be the most challenging part, but shouldn't be impossible... maybe for I'm still a little concerned with how different this is from how count is handled on resources, but I think that could be okay as long as we can make sure the user experience is consistent. |
Yeah I was starting to implement something similar as resources but for all the issues we discussed before it's not going to work. The idea was also to avoid bringing to many changes here. |
Supporting This means that, for example, in ConfigTransformer, before added a node to the Graph we will need to "hack" the path. Seems tricky and error prone :/ |
@apparentlymart I've pushed an update but I don't think I'm in the right direction here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a plausible starting point for the implementation path we discussed. Nice work!
I want to clarify that the design for this feature is not fully fleshed out yet, so I would consider the first goal to do some prototyping and learn more about the problem, rather than to implement a final solution.
If you're motivated to keep working on this I'm certainly happy to keep working with you, but I want to make sure we're aligned on where we are with this right now as I want to be respectful of your time; prototyping of these bigger features can be a rather time-consuming effort.
What you've done so far here has already helped a lot in evaluating a few different options and seeing how this feature might fit into Terraform, so thank you very much for that!
for j := 0; j < len(m.Path()); j++ { | ||
path[j] = m.Path()[j] | ||
} | ||
path[len(path)-1] = strconv.Itoa(m.Count()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should m.Count()
here actually be i
?
Don't have much time this week but I'm definitively still motivated to go through that thing :) |
Hi there, I currently don't have time to work on this. So I will close this PR. I will probably be able to work on that in few weeks but I'm sure someone will tackle that before me (I hope so actually :D) |
Thanks for your work here @Pryz! It was a good prototyping exercise and I'm sure we can build on this work. |
What ever happened to this? Even the 0/1 use case would be beneficial. The main thread at #953 seems to have gone dead since it was locked :( |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
The intent of this PR is to support
count
on modules. This is a work in progress. I will update the description asap with more details.Current status : handle only count = 1 or 0.
Fixing : #953
Todo :
count = "0"