Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add support for the Pulumi Automation API #167

Merged
merged 63 commits into from
Aug 31, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
4bcc0db
feat: add Pulumi Automation API scripts with only AWS support
dekobon Apr 8, 2022
23c1bdc
feat: integrate Automation API scripts with k8s secrets
dekobon May 19, 2022
14c494b
feat: improve authentication error output
dekobon May 20, 2022
27904a6
feat: add differing behavior for container push per provider
dekobon Jun 9, 2022
bfb719e
docs: update examples to reflect changes to secrets
dekobon Jun 9, 2022
8ada509
feat: add support for Digital Ocean to automation API scripts
dekobon Jun 9, 2022
8e36a6c
feat: add support for Digital Ocean Container Registry
dekobon Jun 9, 2022
30e9937
chore: upgrade default helm chart version
dekobon Jun 16, 2022
25fff6b
chore: upgrade example version for ingress image name
dekobon Jun 16, 2022
a9c996a
refactor: break headers into separate file
dekobon Jun 16, 2022
5410bc7
refactor: normalize PulumiProject path property naming
dekobon Jun 16, 2022
4ce3b4e
refactor: on_success closure params to single class
dekobon Jun 16, 2022
13fc758
refactor: separate namespace creation from ingress controller
dekobon Jun 16, 2022
c29e386
refactor: add DO Registry credentials to k8s secrets via a project
dekobon Jun 16, 2022
4539e0a
refactor: make pulumi color settings a method on EnvConfig
dekobon Jun 16, 2022
bad95f9
refactor: add container registry implementation name method
dekobon Jun 16, 2022
1dca513
fix: AWS registry not being referenced using 'repository'
dekobon Jun 16, 2022
028a1ad
bugfix: change pipenv install to pipenv sync to avoid updating deps a…
4141done Jun 3, 2022
8ddf29a
chore: deprecated convenience scripts and projects (#159)
Jun 7, 2022
eadc250
fix: typo in find command was causing pulumi stacks to not be deleted…
Jun 9, 2022
1c5ba81
chore: jenkins fixes and general cleanup of jenkinsfiles (#161)
Jun 15, 2022
cab200d
refactor: improve naming and fix typos
dekobon Jun 16, 2022
b349cb2
feat: install Digital Ocean CLI tool
dekobon Jun 16, 2022
1725755
feat: prompt user for parameters when starting up DO
Jun 15, 2022
3eb6d3e
feat: add dns record support to Digital Ocean provider
dekobon Jun 21, 2022
8907fd3
refactor: change name of container registry credentials project
dekobon Jun 22, 2022
e3fdb6e
docs: small comment addition and doc change
dekobon Jul 11, 2022
c685928
chore: double Helm timeout for Prometheus install
dekobon Jul 11, 2022
a83c758
feat: allow adding new clusters to the kubectl config
dekobon Jul 11, 2022
3f03e56
refactor: add method to insert project in exec order
dekobon Jul 11, 2022
7e69688
chore: simplify function name
dekobon Jul 11, 2022
7045ecd
feat: add check for empty configuration file
dekobon Jul 11, 2022
5ec7fa2
refactor: break apart read and prompt operations
dekobon Jul 11, 2022
b198edc
chore: change encoding to ascii because it is correct
dekobon Jul 11, 2022
c3d3e56
feat: add automation API support for LKE and Harbor on Linode
dekobon Jul 11, 2022
ef2ffb2
refactor: remove bash provision and destroy scripts
dekobon Jul 11, 2022
9d96a9a
fix: typo in function parameter
dekobon Jul 13, 2022
82de087
docs: add additional code comments
dekobon Jul 14, 2022
138157c
feat: migrate to python logging for output
dekobon Jul 15, 2022
c237140
docs: add MARA Runner design document
dekobon Jul 15, 2022
023b4d6
fix: change sed flag from -r to posix compat -E
dekobon Jul 21, 2022
1d916ae
feat: add easy runner script to MARA automation
dekobon Jul 21, 2022
3d357d6
fix: properly pass parameters to main.py from runner
dekobon Jul 21, 2022
e436629
refactor: improve error messaging and operations parsing
dekobon Jul 22, 2022
5ef8665
fix: fixes object has no attribute '__debug_logger_func' error
dekobon Jul 22, 2022
24ee0c0
test: fix test runner to work after directory refactor
dekobon Jul 22, 2022
3a7ca2b
fix: enable force delete on ECR so that MARA can remove it
dekobon Jul 26, 2022
4f5dacd
fix: sensible message on refresh run without prev deployment
dekobon Jul 26, 2022
03de8c8
fix: change import for better compatibility with test runner
dekobon Jul 26, 2022
78468db
chore: upgrade pulumi deps and kubectl versions
dekobon Jul 26, 2022
beeb2f4
fix: add in updates to automation-api branch for version bumps (#172)
Aug 2, 2022
01a563e
chore: additional bug fixes and usability fixes to automation branch …
Aug 3, 2022
3ea8c41
chore: updates to fix digital ocean deployment (#177)
Aug 4, 2022
b20fb64
fix: provide ability to still run kubeconfig deploys until they are c…
Aug 8, 2022
c90d017
fix: added gitkeep for IC manifests dir which is required for repo-on…
qdzlug Aug 8, 2022
bf66271
fix: updated jenkinsfiles for automation api work (#181)
Aug 8, 2022
c70959a
chore: doc updates for automation-api changes (#183)
qdzlug Aug 12, 2022
097589a
fix: adding updates to jenkinsfiles (#185)
qdzlug Aug 15, 2022
2114849
fix: add wheel back into setup_venv.sh (#187)
qdzlug Aug 16, 2022
4934c5b
fix: add variable for password fields in jenkins runs / accept stack …
Aug 30, 2022
181c439
merge: resolved conflicts
Aug 30, 2022
cfee4be
fix: linode jenkinsfile and log level adjust / comment add (#194)
Aug 30, 2022
c5515a3
feat: add new logo to repo (#195)
Aug 30, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ A clear and concise description of what you want to happen.
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Add any other context or screenshots about the feature request here.
6 changes: 3 additions & 3 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
### Proposed changes
Describe the use case and detail of the change. If this PR addresses an issue
on GitHub, make sure to include a link to that issue here in this description
Describe the use case and detail of the change. If this PR addresses an issue
on GitHub, make sure to include a link to that issue here in this description
(not in the title of the PR).

### Checklist
Before creating a PR, run through this checklist and mark each as complete.

- [ ] I have written my commit messages in the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format.
- [ ] I have written my commit messages in the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format.
- [ ] I have read the [CONTRIBUTING](/CONTRIBUTING.md) doc
- [ ] I have added tests (when possible) that prove my fix is effective or that my feature works
- [ ] I have checked that all unit tests pass after adding my changes
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,9 @@ override.tf.json

# End of https://www.toptal.com/developers/gitignore/api/python,pycharm+all,terraform

# Ignore locally installed pyenv environment
.pyenv

*.pyc
!/extras/jwt.token
/pulumi/python/tools/common/config/*.yaml
Expand Down
49 changes: 49 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
- id: check-yaml
args: [--allow-multiple-documents]
- id: check-added-large-files
- id: check-merge-conflict
- id: detect-private-key
- id: trailing-whitespace
- id: mixed-line-ending
- id: end-of-file-fixer
- id: debug-statements
- id: check-merge-conflict
- id: check-ast

- repo: https://github.com/pre-commit/mirrors-autopep8
rev: v1.7.0
hooks:
- id: autopep8

- repo: https://github.com/asottile/dead
rev: v1.5.0
hooks:
- id: dead

- repo: https://github.com/jumanjihouse/pre-commit-hooks
rev: 3.0.0
hooks:
- id: shellcheck
- id: shfmt
- id: markdownlint

- repo: https://github.com/PyCQA/flake8
rev: 5.0.4
hooks:
- id: flake8

- repo: https://github.com/zricethezav/gitleaks
rev: v8.11.0
hooks:
- id: gitleaks

- repo: https://github.com/Yelp/detect-secrets
rev: v1.3.0
hooks:
- id: detect-secrets
12 changes: 7 additions & 5 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,13 +117,15 @@ the community.

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html](https://www.contributor-covenant.org/version/2/0/code_of_conduct.html)
.

Community Impact Guidelines were inspired by [Mozilla's code of conduct
enforcement ladder](https://github.com/mozilla/diversity).
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).

[homepage]: https://www.contributor-covenant.org

For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
[https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq)
. Translations are available at
[https://www.contributor-covenant.org/translations](https://www.contributor-covenant.org/translations).
53 changes: 36 additions & 17 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
# Contributing Guidelines

The following is a set of guidelines for contributing. We really appreciate that you are considering contributing!
The following is a set of guidelines for contributing. We really appreciate
that you are considering contributing!

#### Table Of Contents
## Table Of Contents

[Ask a Question](#ask-a-question)

[Contributing](#contributing)

[Style Guides](#style-guides)
* [Git Style Guide](#git-style-guide)
* [Go Style Guide](#go-style-guide)

* [Git Style Guide](#git-style-guide)
* [Go Style Guide](#go-style-guide)

[Code of Conduct](https://github.com/nginxinc/nginx-wrapper/blob/master/CODE_OF_CONDUCT.md)

Expand All @@ -22,33 +24,50 @@ Please open an Issue on GitHub with the label `question`.

### Report a Bug

To report a bug, open an issue on GitHub with the label `bug` using the available bug report issue template. Please ensure the issue has not already been reported.
To report a bug, open an issue on GitHub with the label `bug` using the
available bug report issue template. Please ensure the issue has not already
been reported.

### Suggest an Enhancement

To suggest an enhancement, please create an issue on GitHub with the label `enhancement` using the available feature issue template.
To suggest an enhancement, please create an issue on GitHub with the label
`enhancement` using the available feature issue template.

### Open a Pull Request

* Fork the repo, create a branch, submit a PR when your changes are tested and ready for review.
* Fork the repo, create a branch, submit a PR when your changes are tested and
ready for review.
* Fill in [our pull request template](/.github/PULL_REQUEST_TEMPLATE.md)

Note: if you’d like to implement a new feature, please consider creating a feature request issue first to start a discussion about the feature.
Note: if you’d like to implement a new feature, please consider creating a
feature request issue first to start a discussion about the feature.

## Style Guides

### Git Style Guide

* Keep a clean, concise and meaningful git commit history on your branch, rebasing locally and squashing before submitting a PR
* Use the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format when writing a commit message, so that changelogs can be automatically generated
* Follow the guidelines of writing a good commit message as described [here](https://chris.beams.io/posts/git-commit/) and summarised in the next few points
* In the subject line, use the present tense ("Add feature" not "Added feature")
* In the subject line, use the imperative mood ("Move cursor to..." not "Moves cursor to...")
* Limit the subject line to 72 characters or less
* Reference issues and pull requests liberally after the subject line
* Add more detailed description in the body of the git message (`git commit -a` to give you more space and time in your text editor to write a good message instead of `git commit -am`)
* Keep a clean, concise and meaningful git commit history on your branch,
rebasing locally and squashing before submitting a PR
* Use the
[Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) format
when writing a commit message, so that changelogs can be automatically
generated
* Follow the guidelines of writing a good commit message as described
[here](https://chris.beams.io/posts/git-commit/) and summarised in the next
few points
* In the subject line, use the present tense
("Add feature" not "Added feature")
* In the subject line, use the imperative mood ("Move cursor to..." not
"Moves cursor to...")
* Limit the subject line to 72 characters or less
* Reference issues and pull requests liberally after the subject line
* Add more detailed description in the body of the git message (
`git commit -a` to give you more space and time in your text editor to
write a good message instead of `git commit -am`)

### Code Style Guide

* Python code should conform to the [PEP-8 style guidelines](https://www.python.org/dev/peps/pep-0008/) whenever possible.
* Python code should conform to the
[PEP-8 style guidelines](https://www.python.org/dev/peps/pep-0008/)
whenever possible.
* Where feasible, include unit tests.
86 changes: 50 additions & 36 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,47 @@
# NGINX Modern Reference Architectures

## Current Test Status

[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B5618%2Fgit%40github.heygears.com%3Anginxinc%2Fkic-reference-architectures.git.svg?type=shield)](https://app.fossa.com/projects/custom%2B5618%2Fgit%40github.heygears.com%3Anginxinc%2Fkic-reference-architectures.git?ref=badge_shield)
![AWS Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_aws_prod&subject=AWS)
![DO Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_do_prod&subject=DigitalOcean)
![LKE Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_lke_prod&subject=Linode)
![AWS Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_aws_prod&subject=AWS)
![DO Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_do_prod&subject=DigitalOcean)
![LKE Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_lke_prod&subject=Linode)
![K3s Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_k3s_prod&subject=K3s)
![MicroK8s Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_mk8s_prod&subject=MicroK8s)
![Minikube Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_minikube_prod&subject=Minikube)
![Minikube Status](https://jenkins.mantawang.com/buildStatus/icon?job=mara_minikube_prod&subject=Minikube)

# NGINX Modern Reference Architectures
![MARA Project](./docs/NGINX-MARA-icon.png)

This repository has the basics for a common way to deploy and manage modern apps. Over time, we'll build more example
architectures using different deployment models and options – including other clouds – and you’ll be able to find those
here.
This repository has the basics for a common way to deploy and manage modern
apps. Over time, we'll build more example architectures using different
deployment models and options – including other clouds – and you’ll be able
to find those here.

## Nomenclature

Internally, we refer to this project as MARA for Modern Application Reference Architecture. The current repository name
reflects the humble origins of this project, as it was started with the purpose of allowing users to build custom
versions of the NGINX Ingress Controller in Kubernetes. This went so well that we expanded it to the project you're
currently viewing.
Internally, we refer to this project as MARA for Modern Application Reference
Architecture. The current repository name reflects the humble origins of this
project, as it was started with the purpose of allowing users to build custom
versions of the NGINX Ingress Controller in Kubernetes. This went so well that
we expanded it to the project you're currently viewing.

## Modern App Architectures

We define modern app architectures as those driven by four characteristics:
*scalability*, *portability*, *resiliency*, and *agility*. While many different aspects of a modern architecture exist,
these are fundamental.
*scalability*, *portability*, *resiliency*, and *agility*. While many different
aspects of a modern architecture exist, these are fundamental.

* **Scalability** – Quickly and seamlessly scale up or down to accommodate spikes or reductions in demand, anywhere in
the world.
* **Scalability** – Quickly and seamlessly scale up or down to accommodate
spikes or reductions in demand, anywhere in the world.

* **Portability** – Easy to deploy on multiple types of devices and infrastructures, on public clouds, and on premises.
* **Portability** – Easy to deploy on multiple types of devices and
infrastructures, on public clouds, and on premises.

* **Resiliency** – Can fail over to newly spun‑up clusters or virtual environments in different availability regions,
clouds, or data centers.
* **Resiliency** – Can fail over to newly spun‑up clusters or virtual
environments in different availability regions, clouds, or data centers.

* **Agility** – Ability to update through automated CI/CD pipelines with higher code velocity and more frequent code
pushes.
* **Agility** – Ability to update through automated CI/CD pipelines with higher
code velocity and more frequent code pushes.

This diagram is an example of what we mean by a **modern app architecture**:
![Modern Apps Architecture Example Diagram](docs/DIAG-NGINX-ModernAppsRefArch-NGINX-MARA-1-0-blog-1024x800.png)
Expand All @@ -53,26 +60,32 @@ To satisfy the four key characteristics, many modern app architectures employ:

For details on the current state of this project, please see the
[readme](pulumi/python/README.md) in the [`pulumi/python`](pulumi/python)
subdirectory. This project is under active development, and the current work is using [Pulumi](https://www.pulumi.com/)
with Python. Additionally, please see
[Status and Issues](docs/status-and-issues.md) for the project's up-to-date build status and known issues.

Subdirectories contained within the root directory separate reference architectures by infrastructure deployment tooling
with additional subdirectories as needed. For example, Pulumi allows the use of multiple languages for deployment. As we
decided to use Python in our first build, there is a `python` subdirectory under the `pulumi` directory.

This project was started to provide a complete, stealable, easy to deploy, and standalone example of how a modern app
architecture can be built. It was driven by the necessity to be flexible and not require a long list of dependencies to
get started. It needs to provide examples of tooling used to build this sort of architecture in the real world. Most
importantly, it needs to work. Hopefully this provides a ‘jumping off’ point for someone to build their own
subdirectory. This project is under active development, and the current work is
using [Pulumi](https://www.pulumi.com/) with Python. Additionally, please see
[Status and Issues](docs/status-and-issues.md) for the project's up-to-date
build status and known issues.

Subdirectories contained within the root directory separate reference
architectures by infrastructure deployment tooling with additional
subdirectories as needed. For example, Pulumi allows the use of multiple
languages for deployment. As we decided to use Python in our first build, there
is a `python` subdirectory under the `pulumi` directory.

This project was started to provide a complete, stealable, easy to deploy, and
standalone example of how a modern app architecture can be built. It was driven
by the necessity to be flexible and not require a long list of dependencies to
get started. It needs to provide examples of tooling used to build this sort of
architecture in the real world. Most importantly, it needs to work. Hopefully
this provides a ‘jumping off’ point for someone to build their own
infrastructure.

## Deployment Tools

### Pulumi

[Pulumi](https://www.pulumi.com/) is a modern Infrastructure as Code (IaC) tool that allows you to write code (node,
Python, Go, etc.) that defines cloud infrastructure. Within the [`pulumi`](pulumi) folder are examples of the pulumi
[Pulumi](https://www.pulumi.com/) is a modern Infrastructure as Code (IaC) tool
that allows you to write code (node, Python, Go, etc.) that defines cloud
infrastructure. Within the [`pulumi`](pulumi) folder are examples of the pulumi
being used to stand up MARA.

## Contribution
Expand All @@ -87,6 +100,7 @@ All code in this repository is licensed under the
[Apache License v2 license](LICENSE).

Open source license notices for all projects in this repository can be
found [here](https://app.fossa.com/reports/92595e16-c0b8-4c68-8c76-59696b6ac219).
found
[here](https://app.fossa.com/reports/92595e16-c0b8-4c68-8c76-59696b6ac219).

[![FOSSA Status](https://app.fossa.com/api/projects/custom%2B5618%2Fgit%40github.heygears.com%3Anginxinc%2Fkic-reference-architectures.git.svg?type=large)](https://app.fossa.com/projects/custom%2B5618%2Fgit%40github.heygears.com%3Anginxinc%2Fkic-reference-architectures.git?ref=badge_large)
52 changes: 25 additions & 27 deletions bin/aws_write_creds.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,40 +2,38 @@
set -o errexit # abort on nonzero exit status
set -o pipefail # don't hide errors within pipes

#
# This script is temporary until we rewrite the AWS deployment following #81 and #82.
# We look into the environment and if we see environment variables for the AWS
# authentication process we move them into a credentials file. This is primarily being
# done at this time to support Jenkins using env vars for creds
#
# This script is temporary until we rewrite the AWS deployment following
# 81 and #82. # We look into the environment and if we see environment
# variables for the AWS # authentication process we move them into a
# credentials file. This is primarily being # done at this time to support
# Jenkins using env vars for creds
#

aws_auth_vars=(AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN)

missing_auth_vars=()
for i in "${aws_auth_vars[@]}"
do
test -n "${!i:+y}" || missing_vars+=("$i")
for i in "${aws_auth_vars[@]}"; do
test -n "${!i:+y}" || missing_vars+=("$i")
done

if [ ${#missing_auth_vars[@]} -ne 0 ]
then
echo "Did not find values for:"
printf ' %q\n' "${missing_vars[@]}"
echo "Will assume they are in credentials file or not needed"
if [ ${#missing_auth_vars[@]} -ne 0 ]; then
echo "Did not find values for:"
printf ' %q\n' "${missing_vars[@]}"
echo "Will assume they are in credentials file or not needed"
else
echo "Creating credentials file"
# Create the directory....
mkdir -p ~/.aws
CREDS=~/.aws/credentials
echo "[default]" > $CREDS
echo "aws_access_key_id=$AWS_ACCESS_KEY_ID" >> $CREDS
echo "aws_secret_access_key=$AWS_SECRET_ACCESS_KEY" >> $CREDS
# This is if we have non-temp credentials...
if [[ -z "${AWS_SESSION_TOKEN+x}" ]]; then
echo "Variable AWS_SESSION_TOKEN was unset; not adding to credentials"
else
echo "aws_session_token=$AWS_SESSION_TOKEN" >> $CREDS
fi
echo "Creating credentials file"
# Create the directory....
mkdir -p ~/.aws
CREDS=~/.aws/credentials
echo "[default]" >$CREDS
echo "aws_access_key_id=$AWS_ACCESS_KEY_ID" >>$CREDS
echo "aws_secret_access_key=$AWS_SECRET_ACCESS_KEY" >>$CREDS
# This is if we have non-temp credentials...
if [[ -z "${AWS_SESSION_TOKEN+x}" ]]; then
echo "Variable AWS_SESSION_TOKEN was unset; not adding to credentials"
else
echo "aws_session_token=$AWS_SESSION_TOKEN" >>$CREDS
fi

fi

Loading