Failure with utf-8 characters on Windows

[cmbrad]

Description :

On windows the default system encoding is cp1252

>> locale.getpreferredencoding()
'cp1252'

as opposed to my RHEL system where:

>> locale.getpreferredencoding()
'UTF-8'

When a plan contains UTF-8 characters, perhaps in a description field for a variable, this causes an error on Windows of the form: 'charmap' codec can't decode byte 0x9d in position 99999: character maps to <undefined>.

To Reproduce

1. Have a variable with a utf-8 specific character such

variable "my_var" {
  description = "My weird characters: “Java” or “Windows”"
  default     = null
}

2. terraform-compliance -p plan.out -f rules
3. Running using python package

ERROR: C:\Users\...\plan.out.json is not a valid JSON file.

       Did you try to convert the binary plan file to json with "terraform show -json C:\Users\...\plan.out.json > C:\Users\...\plan.out.json.json" ?

Expected behavior :

It doesn't crash with an error!

Tested versions :

• terraform-compliance version v1.0.56
• terraform version v0.12.15
• python version 3.7.3

Additional context

Adding 'encoding="utf8"' to the various open calls fixes it - ie open(values, 'r', encoding='utf8') although not sure what impact this has on other versions of Python. For now the workaround is to only run this in CI & stop people adding utf8 characters to modules.

[eerkunt]

Trying to reproduce the problem, once reproduced will fix it permanently.

Meanwhile, did you try to run terraform-compliance after you set PYTHONIOENCODING environment varialbe to UTF-8 ?

[eerkunt]

This is a very good finding 🎉 Thanks @cmbrad !

It looks like PYTHONIOENCODING doesn't solve this problem. Will release 1.0.57 after CI checks pass.

[eerkunt]

Please have a try with 1.0.57 and let me know if this is still not working for you ?

Thanks!

[eerkunt]

Closing the issue due to inactivity. Please do not hesitate to open a new issue if the problem still continue.

Thanks 🎉