Consider getting provider schema using Terraform CLI

[turkenh]

What problem are you facing?

Today, we are generating CRD schemas by importing the package with the schema definition from Terraform provider codebase. While this worked well so far, we should consider breaking this dependency because:

• Terraform documentation explicitly mentions that importing Provider as Go Modules is an unsupported way to interact.
• We only need this at code generation time and do not have any dependency at runtime.
• Some Terraform providers have the schema under internal directory which prevents importing them as is (and needs some workarounds)

How could Terrajet help solve your problem?

As suggested in the Terraform docs, we can just use the Terraform CLI to fetch the schema of the Provider and break go module dependency.

Some projects just care about the schema and resources a provider presents. As of Terraform 0.12, the terraform providers schema -json command can be used to export a JSON representation of the schemas for the providers used in a workspace.

[muvaf]

FYI, I noticed that the schema struct doesn't seem to be using json tags that would allow us to easily unmarshal/marshal.

[turkenh]

FYI, I noticed that the schema struct doesn't seem to be using json tags that would allow us to easily unmarshal/marshal.

@muvaf could you elaborate?

terraform providers schema -json returns a json of ProviderSchemas struct from hashicorp/terraform-json repository. This struct keeps resource schemas with different types than the types in hashicorp/terraform-plugin-sdk which we are currently using in Terrajet code generation pipeline.

See: aws-provider-schema.json.txt

So, if we would like to go with this approach, it looks like we would need to make changes in types package such that it uses types in hashicorp/terraform-json instead.
For most cases this would be quite straightforward since the flow wouldn't change much, however, there are some differences which we would need to handle. One example is, in terraform-json, indeed in go-cty which it uses as for types, there is no int/float but only numbers.

[muvaf]

could you elaborate?

The types in plugin-sdk don't have json tags in their fields, hence, we can't call json.Unmarshal on the output of terraform providers schema -json.

For most cases this would be quite straightforward since the flow wouldn't change much, however, there are some differences which we would need to handle. One example is, in terraform-json, indeed in go-cty which it uses as for types, there is no int/float but only numbers.

I thought maybe we can have a conversion similar to what we do with plugin sdk v1 schema but if the difference is small, we can just change the underlying data type in types package to this one as you suggested and even remove v1 conversion since the input will always be the same no matter what version of plugin sdk provider uses. If you decide to do that, feel free to close this issue since it's mostly for discovery.

[turkenh]

could you elaborate?

The types in plugin-sdk don't have json tags in their fields, hence, we can't call json.Unmarshal on the output of terraform providers schema -json.

Ah ok, I am confused since the output of terraform providers schema -json does not actually contain plugin-sdk types as mentioned.

For most cases this would be quite straightforward since the flow wouldn't change much, however, there are some differences which we would need to handle. One example is, in terraform-json, indeed in go-cty which it uses as for types, there is no int/float but only numbers.

I thought maybe we can have a conversion similar to what we do with plugin sdk v1 schema but if the difference is small, we can just change the underlying data type in types package to this one as you suggested and even remove v1 conversion since the input will always be the same no matter what version of plugin sdk provider uses. If you decide to do that, feel free to close this issue since it's mostly for discovery.

I would prefer to completely remove plugin sdk dependency (in favor of hashicorp/terraform-json) if our input will always be the cli output. However, a conversion as you mentioned might give us the opportunity to gradually experiment that approach, e.g. new ones will use cli output by default but existing ones can migrate later, which sounds like a good approach 👍
I'll spend investigate this a bit and then close this issue in favor of an implementation one.

[turkenh]

I have opened a draft PR for this and create a follow up issue: #175

Closing this one since this is mostly for investigating the approach.