Skip to content

fix: add variable for password fields in jenkins runs / accept stack argument #188

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 43 commits into from
Aug 30, 2022

Conversation

qdzlug
Copy link
Contributor

@qdzlug qdzlug commented Aug 17, 2022

Proposed changes

Fixes a typo in the jenkinsfile, along with ensuring that passwords are all handled via a jenkins variable.

Adds in logic to accept the pulumi stack on the cli and manage the environment file.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have written my commit messages in the Conventional Commits format.
  • I have read the CONTRIBUTING doc
  • I have added tests (when possible) that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation
  • I have rebased my branch onto master
  • I will ensure my PR is targeting the master branch and pulling from my branch from my own fork

Jason Schmidt and others added 30 commits August 1, 2022 17:37
@qdzlug qdzlug changed the title fix: add variable for password fields in jenkins runs fix: add variable for password fields in jenkins runs / accept stack argument Aug 18, 2022
@qdzlug qdzlug requested a review from dekobon August 18, 2022 19:46
@dekobon
Copy link
Collaborator

dekobon commented Aug 24, 2022

Does this change completely remove the environment file or just add the option for a --stack argument?

@qdzlug
Copy link
Contributor Author

qdzlug commented Aug 24, 2022

This change leaves the environment file intact because it occurred to me that we may want to use the file in the future to pass through environment variables to the program (one thing that comes to mind is docker and the various docker replacements).

The logic honors the stack name in the environment if it matches the stack name passed through, updates the environment file with the stackname if the environment file doesn't exist, and uses the value given as a variable in case they differ from each other.

Copy link
Collaborator

@dekobon dekobon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please break apart the commits for content changes to markdown from formatting changes. If other changes are mixing in with the commit for reformatting, it can really difficult to distinguish it when logging through the git history.

Copy link
Contributor

@4141done 4141done left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments that but not approving since I don't have my head around this nearly enough yet - I'll leave that to @dekobon .

"""
module = importlib.import_module(name=f'providers.{provider_name}')
return module.INSTANCE
def write_env(env_config, stack_name):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional, but if you only need filename consider just passing in filename. If this module exclusively dealt with the env_config object then I could see having functions just take in that object and use fields on it but since this is a general module it would be simpler and less error prone not to force this function to have knowledge of the keys on env_config

env_config = env_config_parser.read()
except FileNotFoundError:
sys.exit(2)
append_env(env_config, stack_name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: if we found a file of the same name but id doesn't contain a stack identifier, would it be simpler to to a write_env instead? That way we always know that the file we wind up with contains the env_config we expect? With append I can see some danger of creating a file that has unexpected stuff.

If there's worry about blowing away some existing config, the file could be backed up before this occurs, or we could prompt the user to let us delete it, or quit the process and deal with it themselves.

I'm lacking a lot of context, but in general my point is that appending to an unknown existing file could create unpredictable edge cases, whereas forcing an explicit write of the incoming config to the file would give us more assurance moving forward that we are working with something known.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point; the case I'm seeing this useful for is one where we decide "we need to pass environment variables into the process through the environment file".

In this case, PULUMI_STACK is just one of the potential variables that would be applied, hence we append. My thought is that if the text gets appended to a file that has garbage in it, the parser will either just ignore it or blow up which should cause the parser to fail.

However, let me test both of the above cases to see if I can get a real answer to those questions.

@qdzlug qdzlug merged commit 4934c5b into nginxinc:automation-api Aug 30, 2022
qdzlug added a commit that referenced this pull request Aug 31, 2022
* feat: add Pulumi Automation API scripts with only AWS support

This change adds Python scripts that use the Pulumi Automation
API to stand up MARA like how the bin/start.sh scripts currently
do.

* feat: integrate Automation API scripts with k8s secrets

This change adds a new Pulumi project named 'secrets' to
MARA. This project is used in conjuction with the Pulumi
Automation API to store secrets using the Kubernetes
secret store so the secrets can be used across Pulumi
projects.

* feat: improve authentication error output

This change outputs the results of the AWS cli command
'aws sts get-caller-identity' more tersely and without
a stacktrace when the command fails.

* feat: add differing behavior for container push per provider

* docs: update examples to reflect changes to secrets

* feat: add support for Digital Ocean to automation API scripts

* feat: add support for Digital Ocean Container Registry

* chore: upgrade default helm chart version

* chore: upgrade example version for ingress image name

* refactor: break headers into separate file

* refactor: normalize PulumiProject path property naming

The naming of the property "root_path" conflicted with the
initialization parameter "path" AS WELL AS the method
"path()". This change normalizes the property names such
that they do not ambiguously overlap.

* refactor: on_success closure params to single class

We want to be able to easily add new references to the state that
can be processed for on_success events. As such, the three variables
passed to those closures has been refactored to a single type which
references the original three variables. This will make adding new
variables easier.

* refactor: separate namespace creation from ingress controller

When using container registry credentials with nginx ingress controller,
one must create the credential secrets for the registry in the same
namespace as the ingress controller. By breaking it apart as a separate
step, it allows us to layer in additional logic (such as adding credentials)
after the namespace has been created and before the ingress controller
has been deployed.

* refactor: add DO Registry credentials to k8s secrets via a project

This change adds a new Pulumi project that gets the authentication
credentials for a Digital Ocean Container Repository, encodes them
as a Kubernetes secret, and then stores the secret in the running
cluster's nginx-ingress namespace.

* refactor: make pulumi color settings a method on EnvConfig

* refactor: add container registry implementation name method

* fix: AWS registry not being referenced using 'repository'

AWS ECR refers to itself as a repository and not a registry, we aim
to keep that naming consistent when referring directly to ECR nouns.

This change fixes a bug where we became over-eager using the word
'registry' instead of the noun 'repository' that is hardcoded in
the ECR stack reference.

* bugfix: change pipenv install to pipenv sync to avoid updating deps at build time (#157)

(cherry picked from commit 01ef1ff)

* chore: deprecated convenience scripts and projects (#159)

* chore: remove non-functional kubevip project

* chore: deprecation of resources as discussed in #155

(cherry picked from commit b4ff561)

* fix: typo in find command was causing pulumi stacks to not be deleted (#160)

(cherry picked from commit 0619d5d)

* chore: jenkins fixes and general cleanup of jenkinsfiles (#161)

* fix: typo in find command was causing pulumi stacks to not be deleted

* fix: formatting and find syntax in jenkins (esc for Groovy)

* fix: formatting and find syntax in jenkins (esc for Groovy)

* chore: clean up the comments a bit

(cherry picked from commit cf655d0)

* refactor: improve naming and fix typos

* feat: install Digital Ocean CLI tool

* feat: prompt user for parameters when starting up DO

* feat: add dns record support to Digital Ocean provider

* refactor: change name of container registry credentials project

* docs: small comment addition and doc change

Reference to the DO CLI is removed from the documentation
because it is installed as part of the setup_venv.sh script.

* chore: double Helm timeout for Prometheus install

* feat: allow adding new clusters to the kubectl config

Allow for adding and merging new clusters into the
users kubectl config. The AWS and Digital Ocean
CLIs do this automatically. However, not all SDKs
nor CLI tools do this. Here we add code that does
this so that the same type of functionality can be
done no matter what the underlying infrastructure
provider is.

* refactor: add method to insert project in exec order

Add a method to allow for the insertion of a project
anywhere in the execution order of a provider.

* chore: simplify function name

* feat: add check for empty configuration file

* refactor: break apart read and prompt operations

* chore: change encoding to ascii because it is correct

* feat: add automation API support for LKE and Harbor on Linode

This adds support for Linode using the Automation API scripts.
In order to provide support, the Harbor Container Registry was
added to the requirements. Now, before starting LKE an instance
of Harbor will be started in a Linode compute instance.

* refactor: remove bash provision and destroy scripts

* fix: typo in function parameter

* docs: add additional code comments

* feat: migrate to python logging for output

* docs: add MARA Runner design document

* fix: change sed flag from -r to posix compat -E

Fixes #168

* feat: add easy runner script to MARA automation

This change adds a bash script that sets up the
virtual environment and invokes the automation/main.py
script. Ideally, this would make running MARA much
easier than it currently is.

* fix: properly pass parameters to main.py from runner

* refactor: improve error messaging and operations parsing

* fix: fixes object has no attribute '__debug_logger_func' error

Fixes #166

There is an issue with methods being named with two leading
underscores being passed correctly to Pulumi's async handlers.
In this change, we rename the method and add a check for
the method's existence.

* test: fix test runner to work after directory refactor

* fix: enable force delete on ECR so that MARA can remove it

* fix: sensible message on refresh run without prev deployment

* fix: change import for better compatibility with test runner

* chore: upgrade pulumi deps and kubectl versions

* fix: add in updates to automation-api branch for version bumps (#172)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* chore: additional bug fixes and usability fixes to automation branch (#174)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* chore: updates to fix digital ocean deployment (#177)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: provide ability to still run kubeconfig deploys until they are cut over to automation api (#179)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: add repo-only IC deploy to support kubeconfig deploys

* fix: modifications to handle kubeconfig deploys for now

* fix: recommission bash scripts to support kubeconfig deploys for now

* fix: added gitkeep for IC manifests dir which is required for repo-only deploy (#180)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: add repo-only IC deploy to support kubeconfig deploys

* fix: modifications to handle kubeconfig deploys for now

* fix: recommission bash scripts to support kubeconfig deploys for now

* fix: gitkeep needed for manifests dir under repo nginx

* fix: updated jenkinsfiles for automation api work (#181)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: add repo-only IC deploy to support kubeconfig deploys

* fix: modifications to handle kubeconfig deploys for now

* fix: recommission bash scripts to support kubeconfig deploys for now

* fix: gitkeep needed for manifests dir under repo nginx

* chore: update jenkinsfiles for automation api

* chore: doc updates for automation-api changes (#183)

* fix: adding updates to jenkinsfiles (#185)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: add repo-only IC deploy to support kubeconfig deploys

* fix: modifications to handle kubeconfig deploys for now

* fix: recommission bash scripts to support kubeconfig deploys for now

* fix: gitkeep needed for manifests dir under repo nginx

* chore: update jenkinsfiles for automation api

* fix: updates to the jenkinsfiles

* chore: doc updates for automation-api changes

* fix: update to docker instance for minikube jenkins

* fix: add wheel back into setup_venv.sh (#187)

* fix: add variable for password fields in jenkins runs / accept stack argument / formatting (#188)

* chore: update cert-manager chart and crds

* chore: update logagent (filebeat) chart version

* chore: update nginx IC to latest chart version

* chore: update prometheus chart to latest version

* chore: update logstore (Elasticsearch) to latest chart versoin

* chore: update observability to new yaml and new chart

* chore: update example config with new values

* fix: remediation of deployment bugs

* fix: removed JWT-only logic from BoS

* fix: remove logic for sirius_host from deprecated jwt deploys

* fix: remove deprecated ingress-repo-only project

* fix: adjust min kubectl version deployed

* fix: refactor digitalocean to docean for variables

* fix: add repo-only IC deploy to support kubeconfig deploys

* fix: modifications to handle kubeconfig deploys for now

* fix: recommission bash scripts to support kubeconfig deploys for now

* fix: gitkeep needed for manifests dir under repo nginx

* chore: update jenkinsfiles for automation api

* fix: updates to the jenkinsfiles

* chore: doc updates for automation-api changes

* fix: update to docker instance for minikube jenkins

* fix: add wheel back into setup_venv.sh

* fix: jenkinsfile updates

* feat: accept stack value on CLI, handle mis-match

* chore: reformat markdown to fit standards

* fix: changes requested in #188

* refactor: break up stack environment logic into fnctions

* refactor: formatting changes to main.py

* fix: address PR comments and formatting

* refactor: formatting fixes

* fix: formatting and PR requested changes

* refactor: bash script cleanup

* chore: remove deprecated testcap script

* fix: shell isn't interpreting the args to pulumi right

* fix: still having weird globbing issues.

* fix: adjust jenkinsfiles for new runner syntax

* fix: linode jenkinsfile and log level adjust / comment add (#194)

* fix: update log level and add comment to clarify print stmt

* fix: add closing braces for Linode Jenkinsfile

* fix: cosmetic fix for || construct

* feat: add new logo to repo (#195)

Co-authored-by: Javier Evans <j.evans@f5.com>
Co-authored-by: Jason Schmidt <j.schmidt@f5.com>
Co-authored-by: Jason Schmidt <qdzlug@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants