Add support for the Pulumi Automation API #167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds Python scripts that use the Pulumi Automation API to stand up MARA like how the bin/start.sh scripts currently do.
This change adds a new Pulumi project named 'secrets' to MARA. This project is used in conjuction with the Pulumi Automation API to store secrets using the Kubernetes secret store so the secrets can be used across Pulumi projects.
This change outputs the results of the AWS cli command 'aws sts get-caller-identity' more tersely and without a stacktrace when the command fails.
The naming of the property "root_path" conflicted with the initialization parameter "path" AS WELL AS the method "path()". This change normalizes the property names such that they do not ambiguously overlap.
We want to be able to easily add new references to the state that can be processed for on_success events. As such, the three variables passed to those closures has been refactored to a single type which references the original three variables. This will make adding new variables easier.
When using container registry credentials with nginx ingress controller, one must create the credential secrets for the registry in the same namespace as the ingress controller. By breaking it apart as a separate step, it allows us to layer in additional logic (such as adding credentials) after the namespace has been created and before the ingress controller has been deployed.
This change adds a new Pulumi project that gets the authentication credentials for a Digital Ocean Container Repository, encodes them as a Kubernetes secret, and then stores the secret in the running cluster's nginx-ingress namespace.
AWS ECR refers to itself as a repository and not a registry, we aim to keep that naming consistent when referring directly to ECR nouns. This change fixes a bug where we became over-eager using the word 'registry' instead of the noun 'repository' that is hardcoded in the ECR stack reference.
* fix: typo in find command was causing pulumi stacks to not be deleted * fix: formatting and find syntax in jenkins (esc for Groovy) * fix: formatting and find syntax in jenkins (esc for Groovy) * chore: clean up the comments a bit (cherry picked from commit cf655d0)
Reference to the DO CLI is removed from the documentation because it is installed as part of the setup_venv.sh script.
Allow for adding and merging new clusters into the users kubectl config. The AWS and Digital Ocean CLIs do this automatically. However, not all SDKs nor CLI tools do this. Here we add code that does this so that the same type of functionality can be done no matter what the underlying infrastructure provider is.
Add a method to allow for the insertion of a project anywhere in the execution order of a provider.
* chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins
This was referenced Aug 19, 2022
This was
linked to
issues
Aug 19, 2022
This was
linked to
issues
Aug 19, 2022
This was referenced Aug 19, 2022
Open
…argument / formatting (#188) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins * fix: add wheel back into setup_venv.sh * fix: jenkinsfile updates * feat: accept stack value on CLI, handle mis-match * chore: reformat markdown to fit standards * fix: changes requested in #188 * refactor: break up stack environment logic into fnctions * refactor: formatting changes to main.py * fix: address PR comments and formatting * refactor: formatting fixes * fix: formatting and PR requested changes * refactor: bash script cleanup * chore: remove deprecated testcap script * fix: shell isn't interpreting the args to pulumi right * fix: still having weird globbing issues. * fix: adjust jenkinsfiles for new runner syntax
|
I think we are ready to merge, once the conflicts are resolved. |
added 2 commits
August 30, 2022 14:00
* fix: update log level and add comment to clarify print stmt * fix: add closing braces for Linode Jenkinsfile * fix: cosmetic fix for || construct
|
Passing in all of the development deployment modes; I am going to perform a handful of tests targeting the new automation logic and then we will merge. |
qdzlug
approved these changes
Aug 31, 2022
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
This PR changes MARA to use the Pulumi Automation API and removes support for the bash script based project startup. By using the Automation API, we get the following benefits:
The following requirements have been added:
This change:
Closes #125
Closes #114
Closes #101
Checklist
Before creating a PR, run through this checklist and mark each as complete.