-
Notifications
You must be signed in to change notification settings - Fork 104
fix: add variable for password fields in jenkins runs / accept stack argument #188
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Conversation
Does this change completely remove the environment file or just add the option for a |
This change leaves the environment file intact because it occurred to me that we may want to use the file in the future to pass through environment variables to the program (one thing that comes to mind is docker and the various docker replacements). The logic honors the stack name in the environment if it matches the stack name passed through, updates the environment file with the stackname if the environment file doesn't exist, and uses the value given as a variable in case they differ from each other. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please break apart the commits for content changes to markdown from formatting changes. If other changes are mixing in with the commit for reformatting, it can really difficult to distinguish it when logging through the git history.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a few comments that but not approving since I don't have my head around this nearly enough yet - I'll leave that to @dekobon .
""" | ||
module = importlib.import_module(name=f'providers.{provider_name}') | ||
return module.INSTANCE | ||
def write_env(env_config, stack_name): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional, but if you only need filename
consider just passing in filename
. If this module exclusively dealt with the env_config
object then I could see having functions just take in that object and use fields on it but since this is a general module it would be simpler and less error prone not to force this function to have knowledge of the keys on env_config
env_config = env_config_parser.read() | ||
except FileNotFoundError: | ||
sys.exit(2) | ||
append_env(env_config, stack_name) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: if we found a file of the same name but id doesn't contain a stack identifier, would it be simpler to to a write_env
instead? That way we always know that the file we wind up with contains the env_config
we expect? With append I can see some danger of creating a file that has unexpected stuff.
If there's worry about blowing away some existing config, the file could be backed up before this occurs, or we could prompt the user to let us delete it, or quit the process and deal with it themselves.
I'm lacking a lot of context, but in general my point is that appending to an unknown existing file could create unpredictable edge cases, whereas forcing an explicit write of the incoming config to the file would give us more assurance moving forward that we are working with something known.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good point; the case I'm seeing this useful for is one where we decide "we need to pass environment variables into the process through the environment file".
In this case, PULUMI_STACK is just one of the potential variables that would be applied, hence we append. My thought is that if the text gets appended to a file that has garbage in it, the parser will either just ignore it or blow up which should cause the parser to fail.
However, let me test both of the above cases to see if I can get a real answer to those questions.
* feat: add Pulumi Automation API scripts with only AWS support This change adds Python scripts that use the Pulumi Automation API to stand up MARA like how the bin/start.sh scripts currently do. * feat: integrate Automation API scripts with k8s secrets This change adds a new Pulumi project named 'secrets' to MARA. This project is used in conjuction with the Pulumi Automation API to store secrets using the Kubernetes secret store so the secrets can be used across Pulumi projects. * feat: improve authentication error output This change outputs the results of the AWS cli command 'aws sts get-caller-identity' more tersely and without a stacktrace when the command fails. * feat: add differing behavior for container push per provider * docs: update examples to reflect changes to secrets * feat: add support for Digital Ocean to automation API scripts * feat: add support for Digital Ocean Container Registry * chore: upgrade default helm chart version * chore: upgrade example version for ingress image name * refactor: break headers into separate file * refactor: normalize PulumiProject path property naming The naming of the property "root_path" conflicted with the initialization parameter "path" AS WELL AS the method "path()". This change normalizes the property names such that they do not ambiguously overlap. * refactor: on_success closure params to single class We want to be able to easily add new references to the state that can be processed for on_success events. As such, the three variables passed to those closures has been refactored to a single type which references the original three variables. This will make adding new variables easier. * refactor: separate namespace creation from ingress controller When using container registry credentials with nginx ingress controller, one must create the credential secrets for the registry in the same namespace as the ingress controller. By breaking it apart as a separate step, it allows us to layer in additional logic (such as adding credentials) after the namespace has been created and before the ingress controller has been deployed. * refactor: add DO Registry credentials to k8s secrets via a project This change adds a new Pulumi project that gets the authentication credentials for a Digital Ocean Container Repository, encodes them as a Kubernetes secret, and then stores the secret in the running cluster's nginx-ingress namespace. * refactor: make pulumi color settings a method on EnvConfig * refactor: add container registry implementation name method * fix: AWS registry not being referenced using 'repository' AWS ECR refers to itself as a repository and not a registry, we aim to keep that naming consistent when referring directly to ECR nouns. This change fixes a bug where we became over-eager using the word 'registry' instead of the noun 'repository' that is hardcoded in the ECR stack reference. * bugfix: change pipenv install to pipenv sync to avoid updating deps at build time (#157) (cherry picked from commit 01ef1ff) * chore: deprecated convenience scripts and projects (#159) * chore: remove non-functional kubevip project * chore: deprecation of resources as discussed in #155 (cherry picked from commit b4ff561) * fix: typo in find command was causing pulumi stacks to not be deleted (#160) (cherry picked from commit 0619d5d) * chore: jenkins fixes and general cleanup of jenkinsfiles (#161) * fix: typo in find command was causing pulumi stacks to not be deleted * fix: formatting and find syntax in jenkins (esc for Groovy) * fix: formatting and find syntax in jenkins (esc for Groovy) * chore: clean up the comments a bit (cherry picked from commit cf655d0) * refactor: improve naming and fix typos * feat: install Digital Ocean CLI tool * feat: prompt user for parameters when starting up DO * feat: add dns record support to Digital Ocean provider * refactor: change name of container registry credentials project * docs: small comment addition and doc change Reference to the DO CLI is removed from the documentation because it is installed as part of the setup_venv.sh script. * chore: double Helm timeout for Prometheus install * feat: allow adding new clusters to the kubectl config Allow for adding and merging new clusters into the users kubectl config. The AWS and Digital Ocean CLIs do this automatically. However, not all SDKs nor CLI tools do this. Here we add code that does this so that the same type of functionality can be done no matter what the underlying infrastructure provider is. * refactor: add method to insert project in exec order Add a method to allow for the insertion of a project anywhere in the execution order of a provider. * chore: simplify function name * feat: add check for empty configuration file * refactor: break apart read and prompt operations * chore: change encoding to ascii because it is correct * feat: add automation API support for LKE and Harbor on Linode This adds support for Linode using the Automation API scripts. In order to provide support, the Harbor Container Registry was added to the requirements. Now, before starting LKE an instance of Harbor will be started in a Linode compute instance. * refactor: remove bash provision and destroy scripts * fix: typo in function parameter * docs: add additional code comments * feat: migrate to python logging for output * docs: add MARA Runner design document * fix: change sed flag from -r to posix compat -E Fixes #168 * feat: add easy runner script to MARA automation This change adds a bash script that sets up the virtual environment and invokes the automation/main.py script. Ideally, this would make running MARA much easier than it currently is. * fix: properly pass parameters to main.py from runner * refactor: improve error messaging and operations parsing * fix: fixes object has no attribute '__debug_logger_func' error Fixes #166 There is an issue with methods being named with two leading underscores being passed correctly to Pulumi's async handlers. In this change, we rename the method and add a check for the method's existence. * test: fix test runner to work after directory refactor * fix: enable force delete on ECR so that MARA can remove it * fix: sensible message on refresh run without prev deployment * fix: change import for better compatibility with test runner * chore: upgrade pulumi deps and kubectl versions * fix: add in updates to automation-api branch for version bumps (#172) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * chore: additional bug fixes and usability fixes to automation branch (#174) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * chore: updates to fix digital ocean deployment (#177) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: provide ability to still run kubeconfig deploys until they are cut over to automation api (#179) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: added gitkeep for IC manifests dir which is required for repo-only deploy (#180) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * fix: updated jenkinsfiles for automation api work (#181) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * chore: doc updates for automation-api changes (#183) * fix: adding updates to jenkinsfiles (#185) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins * fix: add wheel back into setup_venv.sh (#187) * fix: add variable for password fields in jenkins runs / accept stack argument / formatting (#188) * chore: update cert-manager chart and crds * chore: update logagent (filebeat) chart version * chore: update nginx IC to latest chart version * chore: update prometheus chart to latest version * chore: update logstore (Elasticsearch) to latest chart versoin * chore: update observability to new yaml and new chart * chore: update example config with new values * fix: remediation of deployment bugs * fix: removed JWT-only logic from BoS * fix: remove logic for sirius_host from deprecated jwt deploys * fix: remove deprecated ingress-repo-only project * fix: adjust min kubectl version deployed * fix: refactor digitalocean to docean for variables * fix: add repo-only IC deploy to support kubeconfig deploys * fix: modifications to handle kubeconfig deploys for now * fix: recommission bash scripts to support kubeconfig deploys for now * fix: gitkeep needed for manifests dir under repo nginx * chore: update jenkinsfiles for automation api * fix: updates to the jenkinsfiles * chore: doc updates for automation-api changes * fix: update to docker instance for minikube jenkins * fix: add wheel back into setup_venv.sh * fix: jenkinsfile updates * feat: accept stack value on CLI, handle mis-match * chore: reformat markdown to fit standards * fix: changes requested in #188 * refactor: break up stack environment logic into fnctions * refactor: formatting changes to main.py * fix: address PR comments and formatting * refactor: formatting fixes * fix: formatting and PR requested changes * refactor: bash script cleanup * chore: remove deprecated testcap script * fix: shell isn't interpreting the args to pulumi right * fix: still having weird globbing issues. * fix: adjust jenkinsfiles for new runner syntax * fix: linode jenkinsfile and log level adjust / comment add (#194) * fix: update log level and add comment to clarify print stmt * fix: add closing braces for Linode Jenkinsfile * fix: cosmetic fix for || construct * feat: add new logo to repo (#195) Co-authored-by: Javier Evans <j.evans@f5.com> Co-authored-by: Jason Schmidt <j.schmidt@f5.com> Co-authored-by: Jason Schmidt <qdzlug@gmail.com>
Proposed changes
Fixes a typo in the jenkinsfile, along with ensuring that passwords are all handled via a jenkins variable.
Adds in logic to accept the pulumi stack on the cli and manage the environment file.
Checklist
Before creating a PR, run through this checklist and mark each as complete.