Skip to content
This repository was archived by the owner on Jan 31, 2021. It is now read-only.

Commit 220f0d0

Browse files
joshmyersjoshmyers
authored
Initial Implementation (#1)
* Add basic main.tf This will hold our global config to be used in other resources * Add default global variables * Add Postgres Aurora This commit has largely been lifted from our backing services module for Postgres Aurora. It creates a Postgres Aurora cluster and writes the outputs of that to SSM. If no master username / password is given, these are generated as a random strings and saved to SSM using SecureString where necessary. If not postgres_name is given, it defaults to using using the terraform-null-label outputted id. * Add VPC backing-services This is part of terraform-root-modules backing service but I think we will likely want to create the VPC outside of this module and pass it in. I’ve left in for now but we can take out later. * Add Postgres read replica * Add Elasticache Redis backing service * Add Kops metadata module This is used to pull information from our kops cluser by doing a load of data lookups based on some tags. We can then pass things like Kops node security groups around to be used by other modules. * Add AmazonMQ backing service for CodeFresh This commit adds ActiveMQ broker and config for CodeFresh Enterprise. An admin user is created and credentials stored encrypted in SSM. A DNS hostname is created for the ActiveMQ endpoints. Note that unlike the other backing modules in here, AmazonMQ resources are not currently their own module. There are only a handful of resources for this AmazonMQ stuff but we can pull it out into a module if we so wish. The default ActiveMQ config [1] has been added. This is optional in the `aws_mq_broker` but due to Terraform not supporting conditional blocks beyond a basic count, it is a pain to conditionally add this. The schema can be found [2] [1] http://svn.apache.org/repos/asf/activemq/trunk/assembly/src/release/conf/activemq.xml [2] https://s3-us-west-2.amazonaws.com/amazon-mq-docs/XML/amazon-mq-active-mq-5.15.0.xsd * Add docs * Remove RDS Aurora Postgres replica * Remove VPC and subnet modules We will be injecting these resources into an existing VPC * Remove need for kops metadata * Move AmazonMQ into own module * Add EFS * Drop Redis Elasticache version to 3.2.6 3.2.10 doesn’t support encryption, see below: ``` Error creating Elasticache Replication Group: InvalidParameterCombination: Encryption features are not supported for engine version 3.2.10. Please use engine version 3.2.6 ``` * Move aws_mq_broker users into module These should have been in the module that calls this module however there is a Terraform bug [1] meaning passing the list of user maps is failing when trying to use a local value. From the calling module, the below does work: ``` users = [{ "username" = "admin" "password" = "defaultpassword" "groups" = ["admin"] "console_access" = true }] ``` however, using locals as we want to, it does not: ``` users = [{ "username" = “${local.admin_user}” "password" = “${local.admin_password}” "groups" = ["admin"] "console_access" = true }] ``` [1] hashicorp/terraform#12263 * Update docs * Remove deprecated mq_broker_name variable * Pin aws-mq-broker module to 0.1.0 release * Add global enabled variable for whole module We can toggle individual components of the backing services by enabling a specific service, or using the global flag to enable/disable all. * Add s3 bucket to CodeFresh backing services. * Rename node_security_groups to security_groups These maybe the security groups of your kops nodes, they may not, so be more generic. * Add usage to README * Pass only 1 or 2 subnets to mq.tf Running ActiveMQ in ACTIVE_STANDBY_MULTI_AZ mode requires you only supply 2 subnets. Any more and the resource will complain. Similarly you must pass a single subnet if running in SINGLE_INSTANCE mode * Actually use postgres_db_name if we pass it in * Add full example * Remove postgres_name variable This should just be var.name as the postgres name is computed in the RDS cluster module using labels. * Pin mq broker module to latest 0.2.0 release * Remove redis_name as this is calculated in module * Update Redis variable descriptions * overwrite SSM parameter is expected as a boolean * Bump AmazonMQ default instance type mq.t2.micro is likely not what you want in production. * Remove null-label since not being used anymore It is in use in all the submodule we call and we delegate down to them for naming. * Bump aws-efs module To ensure we can use enabled flags on it * Bump aws-s3-bucket to 0.1.0 Since we have the initial PR merged and have cut a release * Remove aws-mq-broker enabled flags Bump aws-mq-broker to 0.3.0, which does not feature enabled flags, so remove them. In its current incarnation the terraform-aws-mq-broker does not support the enabled variable allowing for boolean creation of resources in the module, see [1] for more context. [1] cloudposse/terraform-aws-mq-broker#4 * Add optional EFS VPC and subnet_id variables This commit adds variables to enable overriding what VPC and subnet that EFS runs in. If you don’t provide them, it default back to the `var.vpc_id` and `var.subnet_ids` values. During testing of this module, we found something interesting. We were deploying EFS to the backing services VPC, which is a different VPC to the k8s cluster. Our pods were unable to resolve the DNS endpoint of the EFS cluster, despite there being VPC peering between the two, with DNS lookups between them enabled. AWS documentation [1] states that in this scenario, you have a few options: 1) Use the EFS IP address directly (no thanks) 2) Create a DNS entry in Route53 CNAME’ing to the EFS DNS endpoint in a private hosted zone The underlying EFS module does already create a Route53 DNS entry CNAME to the EFS DNS endpoint, but it isn’t in a private zone. The pod could not resolve the Route53 DNS. Deploying EFS into the _same_ VPC as the k8s nodes worked a treat and finally the pods were able to resolve and mount EFS volumes. [1] https://docs.aws.amazon.com/efs/latest/ug/manage-fs-access-vpc-peering.html * Fix typos * Fix typos * Remove EFS + AmazonMQ from CodeFresh services CodeFresh Enterprise is not compatible with AmazonMQ on the protocol level and EFS will be moved into a general Kops backing service. * Remove Terraform glue variables These should be in the caller module, not this generic module. * Update docs and pin example modules * Update docs to remove TODO and add note on enabled
1 parent fc4483f commit 220f0d0

File tree

10 files changed

+1080
-9
lines changed

10 files changed

+1080
-9
lines changed

‎README.md

+343
Large diffs are not rendered by default.

‎README.yaml

+18-9
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,6 @@ badges:
2929
url: "https://slack.cloudposse.com"
3030

3131
related:
32-
- name: "terraform-null-label"
33-
description: "Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes])"
34-
url: "https://github.com/cloudposse/terraform-null-label"
3532
- name: "terraform-aws-rds-cluster"
3633
description: "Terraform module to provision an RDS Aurora cluster for MySQL or Postgres"
3734
url: "https://github.com/cloudposse/terraform-aws-rds-cluster"
@@ -56,21 +53,33 @@ related:
5653

5754
# Short description of this project
5855
description: |-
59-
Terraform module to provision CodeFresh Enterprise backing services for usage with a kops cluster
56+
Terraform module to provision CodeFresh Enterprise backing services
6057
6158
introduction: |-
6259
The module provisions the following resources:
63-
- AWS Aurora primary
64-
- AWS Aurora read replica
60+
- AWS Aurora Postgresql
6561
- AWS Elasticache Redis
66-
- AWS AmazonMQ (ActiveMQ)
62+
- AWS S3 bucket with associated user and permissions
63+
64+
**NOTE:** This module can be enabled or disabled in entirety using the `enabled` flag or you can override certain services e.g. `s3_enabled = "true"` which will enable only the S3 backing service resources given `enabled = "false"`
6765
6866
# How to use this project
6967
usage: |-
68+
```terraform
69+
module "codefresh_backing_services" {
70+
source = "git::https://github.com/cloudposse/terraform-aws-codefresh-backing-services.git?ref=master"
71+
enabled = "true"
72+
name = "${var.name}"
73+
namespace = "${var.namespace}"
74+
region = "${var.region}"
75+
stage = "${var.stage}"
76+
vpc_id = "${module.vpc.vpc_id}"
77+
subnet_ids = ["${module.subnets.private_subnet_ids}"]
78+
security_groups = ["${module.vpc.vpc_default_security_group_id}"]
79+
}
80+
```
7081
7182
For a complete example, see [examples/complete](examples/complete)
72-
TODO
73-
7483
7584
include:
7685
- "docs/targets.md"

‎aurora-postgres.tf

+182
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# Don't use `admin`
2+
# Read more: <https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html>
3+
# ("MasterUsername admin cannot be used as it is a reserved word used by the engine")
4+
variable "postgres_admin_user" {
5+
type = "string"
6+
description = "Postgres admin user name"
7+
default = ""
8+
}
9+
10+
# Must be longer than 8 chars
11+
# Read more: <https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html>
12+
# ("The parameter MasterUserPassword is not a valid password because it is shorter than 8 characters")
13+
variable "postgres_admin_password" {
14+
type = "string"
15+
description = "Postgres password for the admin user"
16+
default = ""
17+
}
18+
19+
variable "postgres_db_name" {
20+
type = "string"
21+
description = "Postgres database name"
22+
default = ""
23+
}
24+
25+
# db.r4.large is the smallest instance type supported by Aurora Postgres
26+
# https://aws.amazon.com/rds/aurora/#
27+
variable "postgres_instance_type" {
28+
type = "string"
29+
default = "db.r4.large"
30+
description = "EC2 instance type for Postgres cluster"
31+
}
32+
33+
variable "postgres_cluster_size" {
34+
type = "string"
35+
default = "2"
36+
description = "Postgres cluster size"
37+
}
38+
39+
variable "postgres_cluster_enabled" {
40+
type = "string"
41+
default = ""
42+
description = "Set to false to prevent the module from creating any resources"
43+
}
44+
45+
variable "postgres_cluster_family" {
46+
type = "string"
47+
default = "aurora-postgresql9.6"
48+
description = "Postgres cluster DB family. Currently supported values are `aurora-postgresql9.6` and `aurora-postgresql10`"
49+
}
50+
51+
variable "postgres_maintenance_window" {
52+
type = "string"
53+
default = "sun:03:00-sun:04:00"
54+
description = "Weekly time range during which system maintenance can occur, in UTC"
55+
}
56+
57+
locals {
58+
postgres_cluster_enabled = "${var.postgres_cluster_enabled != "" ? var.postgres_cluster_enabled : var.enabled}"
59+
postgres_admin_user = "${length(var.postgres_admin_user) > 0 ? var.postgres_admin_user : join("", random_string.postgres_admin_user.*.result)}"
60+
postgres_admin_password = "${length(var.postgres_admin_password) > 0 ? var.postgres_admin_password : join("", random_string.postgres_admin_password.*.result)}"
61+
postgres_db_name = "${var.postgres_db_name != "" ? var.postgres_db_name : join("", random_pet.postgres_db_name.*.id)}"
62+
}
63+
64+
module "aurora_postgres" {
65+
source = "git::https://github.com/cloudposse/terraform-aws-rds-cluster.git?ref=tags/0.10.0"
66+
namespace = "${var.namespace}"
67+
stage = "${var.stage}"
68+
name = "${var.name}"
69+
attributes = ["postgresql"]
70+
engine = "aurora-postgresql"
71+
cluster_family = "${var.postgres_cluster_family}"
72+
instance_type = "${var.postgres_instance_type}"
73+
cluster_size = "${var.postgres_cluster_size}"
74+
admin_user = "${local.postgres_admin_user}"
75+
admin_password = "${local.postgres_admin_password}"
76+
db_name = "${local.postgres_db_name}"
77+
db_port = "5432"
78+
maintenance_window = "${var.postgres_maintenance_window}"
79+
vpc_id = "${var.vpc_id}"
80+
subnets = ["${var.subnet_ids}"]
81+
zone_id = "${local.zone_id}"
82+
security_groups = ["${var.security_groups}"]
83+
enabled = "${local.postgres_cluster_enabled}"
84+
}
85+
86+
resource "random_pet" "postgres_db_name" {
87+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
88+
separator = "_"
89+
}
90+
91+
resource "random_string" "postgres_admin_user" {
92+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
93+
length = 8
94+
special = false
95+
number = false
96+
}
97+
98+
resource "random_string" "postgres_admin_password" {
99+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
100+
length = 16
101+
special = true
102+
}
103+
104+
resource "aws_ssm_parameter" "aurora_postgres_database_name" {
105+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
106+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_database_name")}"
107+
value = "${module.aurora_postgres.name}"
108+
description = "Aurora Postgres Database Name"
109+
type = "String"
110+
overwrite = "${var.overwrite_ssm_parameter}"
111+
}
112+
113+
resource "aws_ssm_parameter" "aurora_postgres_master_username" {
114+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
115+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_master_username")}"
116+
value = "${module.aurora_postgres.user}"
117+
description = "Aurora Postgres Username for the master DB user"
118+
type = "String"
119+
overwrite = "${var.overwrite_ssm_parameter}"
120+
}
121+
122+
resource "aws_ssm_parameter" "aurora_postgres_master_password" {
123+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
124+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_master_password")}"
125+
value = "${module.aurora_postgres.password}"
126+
description = "Aurora Postgres Password for the master DB user"
127+
type = "SecureString"
128+
key_id = "${data.aws_kms_key.chamber_kms_key.id}"
129+
overwrite = "${var.overwrite_ssm_parameter}"
130+
}
131+
132+
resource "aws_ssm_parameter" "aurora_postgres_master_hostname" {
133+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
134+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_master_hostname")}"
135+
value = "${module.aurora_postgres.master_host}"
136+
description = "Aurora Postgres DB Master hostname"
137+
type = "String"
138+
overwrite = "${var.overwrite_ssm_parameter}"
139+
}
140+
141+
resource "aws_ssm_parameter" "aurora_postgres_replicas_hostname" {
142+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
143+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_replicas_hostname")}"
144+
value = "${module.aurora_postgres.replicas_host}"
145+
description = "Aurora Postgres DB Replicas hostname"
146+
type = "String"
147+
overwrite = "${var.overwrite_ssm_parameter}"
148+
}
149+
150+
resource "aws_ssm_parameter" "aurora_postgres_cluster_name" {
151+
count = "${local.postgres_cluster_enabled == "true" ? 1 : 0}"
152+
name = "${format(var.chamber_format, local.chamber_service, "aurora_postgres_cluster_name")}"
153+
value = "${module.aurora_postgres.cluster_name}"
154+
description = "Aurora Postgres DB Cluster Identifier"
155+
type = "String"
156+
overwrite = "${var.overwrite_ssm_parameter}"
157+
}
158+
159+
output "aurora_postgres_database_name" {
160+
value = "${local.postgres_cluster_enabled == "true" ? module.aurora_postgres.name : ""}"
161+
description = "Aurora Postgres Database name"
162+
}
163+
164+
output "aurora_postgres_master_username" {
165+
value = "${local.postgres_cluster_enabled == "true" ? module.aurora_postgres.user : ""}"
166+
description = "Aurora Postgres Username for the master DB user"
167+
}
168+
169+
output "aurora_postgres_master_hostname" {
170+
value = "${local.postgres_cluster_enabled == "true" ? module.aurora_postgres.master_host : ""}"
171+
description = "Aurora Postgres DB Master hostname"
172+
}
173+
174+
output "aurora_postgres_replicas_hostname" {
175+
value = "${local.postgres_cluster_enabled == "true" ? module.aurora_postgres.replicas_host : ""}"
176+
description = "Aurora Postgres Replicas hostname"
177+
}
178+
179+
output "aurora_postgres_cluster_name" {
180+
value = "${local.postgres_cluster_enabled == "true" ? module.aurora_postgres.cluster_name : ""}"
181+
description = "Aurora Postgres Cluster Identifier"
182+
}

‎docs/targets.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
## Makefile Targets
2+
```
3+
Available targets:
4+
5+
help Help screen
6+
help/all Display help for all targets
7+
help/short This help short screen
8+
lint Lint terraform code
9+
10+
```

‎docs/terraform.md

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
2+
## Inputs
3+
4+
| Name | Description | Type | Default | Required |
5+
|------|-------------|:----:|:-----:|:-----:|
6+
| attributes | Additional attributes (e.g. `1`) | list | `<list>` | no |
7+
| chamber_format | Format to store parameters in SSM, for consumption with chamber | string | `/%s/%s` | no |
8+
| chamber_service | `chamber` service name. See [chamber usage](https://github.com/segmentio/chamber#usage) for more details | string | `` | no |
9+
| delimiter | Delimiter to be used between `namespace`, `stage`, `name` and `attributes` | string | `-` | no |
10+
| enabled | Set to false to prevent the module from creating any resources | string | `true` | no |
11+
| kms_key_id | KMS key ID used to encrypt SSM parameters | string | `` | no |
12+
| name | Name (e.g. `codefresh`) | string | `cf` | no |
13+
| namespace | Namespace (e.g. `eg` or `cp`) | string | - | yes |
14+
| overwrite_ssm_parameter | Whether to overwrite an existing SSM parameter | string | `true` | no |
15+
| postgres_admin_password | Postgres password for the admin user | string | `` | no |
16+
| postgres_admin_user | Postgres admin user name | string | `` | no |
17+
| postgres_cluster_enabled | Set to false to prevent the module from creating any resources | string | `` | no |
18+
| postgres_cluster_family | Postgres cluster DB family. Currently supported values are `aurora-postgresql9.6` and `aurora-postgresql10` | string | `aurora-postgresql9.6` | no |
19+
| postgres_cluster_size | Postgres cluster size | string | `2` | no |
20+
| postgres_db_name | Postgres database name | string | `` | no |
21+
| postgres_instance_type | EC2 instance type for Postgres cluster | string | `db.r4.large` | no |
22+
| postgres_maintenance_window | Weekly time range during which system maintenance can occur, in UTC | string | `sun:03:00-sun:04:00` | no |
23+
| redis_apply_immediately | Whether to apply changes immediately or during the next maintenance window | string | `true` | no |
24+
| redis_at_rest_encryption_enabled | Enable Redis encryption at rest | string | `true` | no |
25+
| redis_auth_token | Auth token for password protecting Redis. `transit_encryption_enabled` must be set to `true`! Password must be longer than 16 chars | string | `` | no |
26+
| redis_automatic_failover | Whether to enable automatic failover | string | `true` | no |
27+
| redis_cluster_enabled | Set to false to prevent the module from creating any resources | string | `` | no |
28+
| redis_cluster_size | Redis cluster size | string | `2` | no |
29+
| redis_engine_version | Version of Redis engine | string | `5.0.0` | no |
30+
| redis_instance_type | EC2 instance type for Redis cluster | string | `cache.t2.medium` | no |
31+
| redis_maintenance_window | Weekly time range during which system maintenance can occur, in UTC | string | `sun:03:00-sun:04:00` | no |
32+
| redis_params | A list of Redis parameters to apply. Note that parameters may differ from a Redis family to another | list | `<list>` | no |
33+
| redis_transit_encryption_enabled | Enable TLS for Redis cluster | string | `true` | no |
34+
| s3_access_key_name | S3 user IAM access key name for storing in SSM. Default to aws_acces_key_id so chamber exports as AWS_ACCESS_KEY_ID, a standard AWS IAM ENV variable | string | `aws_access_key_id` | no |
35+
| s3_allowed_bucket_actions | List of actions to permit for S3 bucket | list | `<list>` | no |
36+
| s3_enabled | Set to false to prevent the module from creating any resources | string | `` | no |
37+
| s3_secret_key_name | S3 user IAM secret key name for storing in SSM. Default to aws_secret_acces_key so chamber exports as AWS_SECRET_ACCESS_KEY, a standard AWS IAM ENV variable | string | `aws_secret_access_key` | no |
38+
| s3_user_enabled | Set to `true` to create an S3 user with permission to access the bucket | string | `` | no |
39+
| s3_versioning_enabled | Whether to enable versioning on the S3 bucket. | string | `false` | no |
40+
| security_groups | List of security groups to be allowed to connect to the CodeFresh backing services | list | `<list>` | no |
41+
| stage | Stage (e.g. `prod`, `dev`, `staging`) | string | - | yes |
42+
| subnet_ids | A list of subnet IDs to launch the CodeFresh backing services in | list | `<list>` | no |
43+
| tags | Additional tags (e.g. map(`Cluster`,`us-east-1.cloudposse.co`) | map | `<map>` | no |
44+
| vpc_id | VPC ID for the CodeFresh backing services | string | - | yes |
45+
| zone_name | DNS zone name | string | - | yes |
46+
47+
## Outputs
48+
49+
| Name | Description |
50+
|------|-------------|
51+
| aurora_postgres_cluster_name | Aurora Postgres Cluster Identifier |
52+
| aurora_postgres_database_name | Aurora Postgres Database name |
53+
| aurora_postgres_master_hostname | Aurora Postgres DB Master hostname |
54+
| aurora_postgres_master_username | Aurora Postgres Username for the master DB user |
55+
| aurora_postgres_replicas_hostname | Aurora Postgres Replicas hostname |
56+
| elasticache_redis_host | Elasticache Redis host |
57+
| elasticache_redis_id | Elasticache Redis cluster ID |
58+
| elasticache_redis_security_group_id | Elasticache Redis security group ID |
59+
| s3_access_key_id | The access key ID |
60+
| s3_bucket_arn | The s3 bucket ARN |
61+
| s3_secret_access_key | The secret access key. This will be written to the state file in plain-text |
62+
| s3_user_arn | The ARN assigned by AWS for the user |
63+
| s3_user_name | Normalized IAM user name |
64+
| s3_user_unique_id | The user unique ID assigned by AWS |
65+

0 commit comments

Comments
 (0)